A procurement assistant can read a request, compare suppliers, and prepare an order. Connecting it to the purchasing system introduces a further question: which commitments can it make on the organisation’s behalf? The answer depends on more than how convincingly it explains its recommendation.
An AI agent governance framework defines that delegated authority and establishes how it is enforced, evaluated, monitored, and withdrawn. For this article, an agent is an AI-enabled application that can select and invoke tools while carrying out a task. Its autonomy may be limited to preparing a proposal or extend to actions that change external systems.
The useful unit of governance is the complete application in its business context. Model behaviour matters, but so do credentials, tool interfaces, approval mechanisms, stored state, and the processes that receive the result. A well-written instruction cannot compensate for an integration that permits an unauthorised payment.
Four Pillars of an AI Agent Governance Framework
Agent governance extends beyond reviewing generated content. An agent can call APIs, execute code, change records, and trigger commitments, so the framework must connect policy to the controls on those actions.
| Pillar | Controls to establish | Evidence to maintain |
|---|---|---|
| Identity, ownership, and accountability | Named business and technical owners, a distinct agent identity, and an approved autonomy level. | Ownership records, identity lifecycle records, and an action-level description of delegated authority. |
| Data and access governance | Least-privilege access to data and tools, isolated environments, and protected memory and state. | Approved permissions, environment boundaries, data provenance, and access-review results. |
| Behavioural guardrails and runtime enforcement | Policy checks before execution, approval for consequential actions, rate limits, circuit breakers, and stop controls. | Policy decisions, action-bound approvals, boundary tests, and containment exercises. |
| Observability, auditing, and evidence | A live agent registry, correlated execution records, monitoring, and controlled configuration changes. | Version history, tool-call outcomes, approval records, incident records, and reconciliation with downstream systems. |
Describe Authority as Specific Actions
Start with the business process owner and a description of what the agent may do. For procurement, distinguish reading approved supplier information, drafting an order, submitting it for review, placing the order, and changing supplier bank details. These actions have different consequences even when one interface exposes them together.
Name both a business owner accountable for the use and a technical owner responsible for its implementation and operation. Record who accepts residual risk, approves changes, and responds to incidents. Ownership should remain clear when a supplier provides the platform or several teams maintain different components.
Authority should specify the relevant conditions. An agent might prepare orders for an assigned cost centre using approved suppliers, while a buyer authorises the commitment. Another deployment might allow routine purchases within a defined category and budget, with exceptions routed for review. Those are design choices that require evidence and an accountable decision.
Classify autonomy by the actions allowed: read-only advice, proposals requiring approval, bounded autonomous execution, or higher-impact transactions requiring additional controls. These are practical categories for the organisation to define, rather than a universal risk classification. A move from drafting an order to placing it needs a new authority decision even if the model stays the same.
Include what happens when the request is ambiguous or the available information conflicts. The agent needs an escalation route and a way to return incomplete work without inventing a resolution. A task objective such as obtaining supplies does not itself authorise every action that could help achieve it.
Document who can change these boundaries. Adding a tool, expanding a spending limit, or allowing a new destination can alter the risk substantially. Treat those changes as changes to delegated authority, with review proportionate to their consequences.
Enforce Permissions Outside the Model
The model can propose an action, but the application and downstream service should determine whether that action is permitted. This separates uncertain interpretation from the checks governing access and commitment.
Use an identifiable execution context with the permissions needed for the task. Where an agent acts for a person, preserve the relevant user and business scope. A shared administrator credential can erase the distinction between what the user may request and what the integration can technically perform.
Give each deployed agent a distinct non-human identity using the platform’s supported workload-identity or identity and access management mechanism. Record the agent identity alongside the initiating user and individual run so actions can be attributed without giving the agent unrestricted use of that person’s credentials. A display name in a chat interface is insufficient for this purpose.
Tool design should narrow the available operations. A function that creates a draft order with validated fields is easier to constrain than a general interface that accepts arbitrary commands against the purchasing database. Validate supplier identifiers, destinations, amounts, and applicable permissions at execution time.
Apply least privilege to specific data partitions, API operations, destinations, and Model Context Protocol (MCP) tools or servers where used. Permission to connect to a server should not imply permission to invoke every tool it exposes. Keep development, testing, and production credentials, state, and data separate; a sandbox should not be able to reach live purchasing records through an inherited production credential.
Route proposed tool calls through a runtime policy check before execution. An application service, policy engine, or agent gateway can validate the tool, arguments, current identity, limits, and required approval, then reject the call if a condition fails. Downstream services should enforce their own authorisation as well. This makes the control effective even when the agent proposes an action that contradicts its instructions.
For an external security reference, OWASP’s 2025 guidance on excessive agency identifies excessive functionality, permissions, and autonomy as sources of damaging actions. It recommends limiting tool capability and enforcing authorisation in downstream systems. Those principles are useful even when a model usually follows its instructions.
Credential lifecycle matters too. Establish how credentials are issued, rotated, restricted, and revoked. Removing a chat session should not leave an independently running workflow with indefinite access. Operational controls need to reach the services that actually carry out the work.
Keep Retrieved Material Separate From Instructions
An agent may encounter supplier messages, attachments, websites, and tool responses while completing its task. Those sources can contain text that resembles an instruction. The application should treat them as material to evaluate within the task, with no inherent authority to change its permissions or objective.
For example, a supplier document might tell the agent to update payment details before creating an order. The text could be a legitimate request, a mistake, or a malicious attempt to redirect the workflow. Whatever its origin, it should not become authorisation to alter the supplier record.
The InjecAgent research paper examines indirect prompt injection in tool-integrated agents through a benchmark. It provides evidence that external content can be used to attempt harmful redirection. Its experimental results do not establish a universal failure rate for every production system.
Layered controls should limit the consequences of such attempts. Separate instructions from retrieved content in the application design, restrict accessible tools and data, validate actions, and require independent approval for sensitive changes. Prompt instructions and content screening can contribute, but they should not be the only barrier protecting a consequential action.
Make Approval About the Actual Commitment
A reviewer needs to see what will happen, why it is proposed, and what evidence supports it. For an order, show the supplier, items, quantities, total cost, delivery destination, relevant budget, and any exceptions. A fluent summary can assist review while leaving the underlying values available for inspection.
Bind approval to the action reviewed. If the agent changes the destination or total after approval, the original decision should not silently authorise the new transaction. The execution path should check the approved values and whether relevant conditions still hold.
Approval design also needs to account for workload. Asking someone to approve hundreds of opaque proposals may produce routine acceptance without meaningful scrutiny. Reduce the scope of automation, improve the evidence shown, or route only well-defined exceptions where that is appropriate and permitted.
Define expiry and rejection behaviour. An order approved before a price or budget change may need reassessment. A rejected proposal should not be resubmitted repeatedly with minor wording changes until someone accepts it. The workflow must preserve the decision that was made.
Govern State, Retries, and Delegation
Agent state includes more than conversation history. It can include retrieved information, intermediate plans, cached results, pending approvals, and records of completed actions. Establish which parts are authoritative, how long they are retained, and how they are isolated between users and tasks.
Stored preferences should not override current access rules. If a buyer moves to another department, an earlier task history should not preserve their former purchasing authority. Permissions need to be checked against current policy when the action occurs.
Protect persistent memory, vector stores, and cached sessions against both cross-user leakage and poisoned content. Scope reads and writes to the appropriate user, tenant, and task; preserve source provenance; and control which information can become a durable preference or instruction. Retrieved text should not silently acquire authority because it has been copied into memory. Include retention, deletion, and correction of stored information in the operating process.
Retries introduce another problem. If an order request times out after the purchasing system accepts it, the agent may be uncertain whether the action succeeded. Repeating the request could create a duplicate. Use operation identifiers, idempotent interfaces where available, and reconciliation with the system of record before retrying an uncertain commitment.
Multi-agent workflows need equally explicit boundaries. A research agent passing a supplier recommendation to a purchasing agent should pass evidence and provenance without granting additional authority. The receiving component must validate the request within its own scope. Delegation should preserve the original constraints, including spending and data-access limits.
The organisation should also limit resource consumption. Set appropriate budgets for execution time, tool calls, and external spend. A workflow that cannot complete should terminate or escalate predictably rather than continue searching indefinitely.
Evaluate the Workflow Through Failure Cases
Evaluation should establish whether the deployed system completes permitted tasks and respects its boundaries. A test of final answer quality will miss an unauthorised intermediate action, even if the agent later produces an acceptable explanation.
For the procurement example, relevant cases include a request outside the user’s cost centre, contradictory supplier records, an instruction embedded in an attachment, an expired approval, a timed-out order submission, and an attempted purchase beyond the permitted limit. Include combinations that are plausible in the actual workflow.
Measure the actions taken as well as the outcome. Did the service reject an unauthorised call? Did the agent expose restricted information before stopping? Did it escalate the uncertain result with enough evidence for a person to resolve it? A blocked harmful action is useful evidence about a control, while an attempted violation may still indicate a behaviour problem worth investigating.
Run representative cases when changing models, prompts, tools, permissions, or orchestration. Record versions and configurations so results can be interpreted. Passing a finite test set supports a bounded release decision; it cannot prove that every future interaction will be safe.
Broader risk assessment can draw on the voluntary NIST AI RMF and its Generative AI Profile. These provide risk-management resources, while the application’s specific controls still need engineering and verification.
Preserve Evidence and Practise Containment
Maintain a live registry of deployed agents. Record each agent’s purpose, business and technical owners, identity, environment, model and application versions, authorised tools, autonomy scope, data sources, and operational status. Update the record when permissions or configurations change and retain the change history. The registry should help responders find every active deployment affected by a vulnerable tool or compromised credential.
Execution records should connect the initiating request, identity, relevant configuration, tool calls, approval decisions, and downstream outcomes. They should make it possible to reconstruct what happened without retaining every piece of sensitive content indefinitely.
Use structured logs and correlated traces, such as OpenTelemetry traces where supported, to connect runs, API calls, policy decisions, retries, and errors across services. Capture the evidence and concise decision summaries needed for review, subject to data minimisation. These records support reconstruction of observable actions; they do not make model behaviour deterministic or provide a reliable account of hidden internal reasoning.
An agent’s explanation is not a substitute for an authoritative transaction record. Reconcile important actions with the purchasing system and record uncertainty when a result cannot be confirmed. Protect logs with appropriate access and retention rules; prompts and tool responses can themselves contain personal or confidential information.
Monitoring needs an operational response. Assign responsibility for unusual spending, repeated denials, duplicate submissions, excessive retries, and unexpected destinations. Specify who can pause new tasks, revoke credentials, cancel queued actions, and investigate affected records.
Distinguish a controlled pause from an emergency stop. A pause can prevent new work while authorised in-flight tasks are reconciled; an emergency stop may also revoke access and block execution at the tool boundary. Use automatic rate limits and circuit breakers to halt repeated failures or runaway multi-step activity before a person responds. Test how these controls behave across queued tasks and delegated agents, including whether restarting could repeat an earlier commitment.
A stop control cannot recall an email already delivered or automatically reverse every purchase. Recovery may require cancellation, compensating transactions, supplier contact, or human review. Test those procedures with the operational team and establish how work continues while the agent is unavailable.
Map Controls to Standards and Applicable Obligations
Use external references to challenge the framework and organise its evidence. Distinguish voluntary risk guidance, management-system requirements, security threat catalogues, and legal obligations rather than treating them as interchangeable certifications.
| Reference | How it contributes to agent governance |
|---|---|
| NIST AI Risk Management Framework | A voluntary structure for governing, mapping, measuring, and managing AI risk across the lifecycle, including assessment before deployment. |
| ISO/IEC 42001 | Requirements for establishing, maintaining, and improving an organisational AI management system. Use it to connect agent controls with policy, accountability, and continual improvement. |
| OWASP Top 10 for Agentic Applications | An agent-specific security reference for threat modelling and testing, including goal hijacking, tool misuse, identity abuse, memory poisoning, and cascading failures. |
| EU AI Act | A legal framework whose applicable requirements depend on the system, organisational role, jurisdictional scope, and relevant application dates. Assess transparency, documentation, and human-oversight obligations for the actual deployment rather than assuming every agent is legally high-risk. |
| New Zealand Public Service Artificial Intelligence Framework | A public-service reference covering principles such as human oversight, privacy and data ethics, transparency, explainability, and accountability. Public agencies should map the proposed use to relevant public-service guidance alongside their legal obligations. |
The references inform different parts of the framework. Document which controls address which risks or obligations, who owns them, and what evidence shows that they operate in the deployed system.
The compliance requirements article considers the legal obligations around the deployment, while the operating-model comparison examines responsibility across teams. Agent governance connects those responsibilities to each permitted action, giving the organisation evidence that delegated work stays within its authority and a practical response when it does not.
