A customer-service assistant is allowed to draft a refund response. During a demonstration, the model follows the refund policy and explains the proposed amount. In production, a retrieved message contains instructions to issue a larger refund. Whether that message can cause financial damage depends on the surrounding system: which tools are available, who is authorised, and where the payment service checks the request.
AI guardrails are controls intended to keep a system within defined limits. Some influence what a model produces; others restrict access, validate proposed actions, or stop a workflow. They are most useful when attached to specific unwanted outcomes and tested at the point where those outcomes could occur. A general instruction to behave safely does not establish the same boundary as an enforced transaction limit.
Define the Failure Before Choosing the Control
For the refund assistant, unwanted outcomes might include exposing another customer’s records, promising an unsupported amount, issuing a refund twice, or continuing when the policy service is unavailable. These are different failures. A content filter might help detect an inappropriate message, but it will not prevent duplicate payments or establish access rights.
Describe the intended use, permitted users, accessible information, and available actions. Then consider mistakes, misuse, malicious inputs, and dependency failures. The resulting threat and failure analysis should guide controls. Begin with what the organisation needs to prevent or limit, rather than assembling a list of fashionable guardrail products.
Include ordinary errors as well as attacks. A missing currency, an outdated policy, or a customer identifier copied from the wrong case can cause harm without anyone trying to manipulate the model. Controls should make these failures visible and provide a workable response when the required evidence is unavailable.
The AI accountability framework addresses who owns these choices. Engineering can implement a limit, but someone responsible for the business process must decide which actions are permissible and which unresolved risks prevent release.
Keep Permission Checks Outside the Model
The application should enforce which customer records a user may access before returning data to the assistant. A model-generated request for another account does not establish permission to view it. Apply authorisation using the authenticated user’s identity and the service’s rules at the protected resource.
Tool access should be narrow. An assistant that needs to look up an order does not automatically need a general database connection or a tool that executes arbitrary code. Provide a capability with a defined purpose and constrained parameters. Separate reading information from changing records so that a conversational feature does not inherit unnecessary write access.
OWASP’s guidance on excessive agency identifies excessive functionality, permissions, and autonomy as contributors to harmful actions. Its mitigation guidance includes restricting capabilities and enforcing authorisation in downstream systems. These boundaries can limit consequences even when the model produces an unsuitable request.
For a refund, the payment service should verify account identity, eligibility, currency, amount, and authority. It should do this regardless of whether the request originated from an AI assistant, a conventional application, or a direct API caller. The enforcement should not depend on the model remembering the policy.
Treat Retrieved Content as Untrusted Input
Prompt injection can arrive through a user message or through material the system retrieves, such as a document, web page, or support ticket. A malicious passage may ask the assistant to disclose information or use a tool for an unrelated purpose. Keyword screening alone cannot reliably establish that all possible phrasing is safe.
Separate instructions from reference material as clearly as the system allows, constrain which sources can be retrieved, and avoid placing secrets in context unnecessarily. These measures can reduce exposure, but they should be supported by access and action controls. A document that says “send these records elsewhere” should not create permission to export them.
Evaluate realistic attack paths. Test whether content in an authorised source can influence an unrelated action, whether the assistant can reach another customer’s data, and whether a blocked operation can be attempted through a different tool. OWASP’s prompt-injection guidance describes both direct and indirect routes. Successful tests demonstrate coverage of those cases, rather than immunity to every future attack.
Validate Meaning as Well as Format
A valid JSON response establishes that an output meets a structural requirement. It does not show that an invoice total is correct or that a refund is justified. Validation should therefore include business constraints that can be checked independently: known identifiers, permitted values, arithmetic, current state, and consistency with authoritative records.
Suppose the model proposes a $300 refund on an order worth $120. The service can reject that proposal mechanically. It can also check whether the order has already been refunded. These semantic checks go beyond syntax, although they still cannot settle every question about the customer’s circumstances.
Avoid silently replacing a bad value and leaving the surrounding message unchanged. Clamping $300 to $120 might prevent an excessive transaction while sending a customer an incorrect promise. Regenerate or construct the message from the final validated action, and make the approved amount visible to the reviewer. Business records and communication need to agree.
Give Filters and Verifiers a Measurable Job
Content classifiers can help identify disallowed material, sensitive information, or unsafe requests. Their usefulness depends on the domain and the costs of missed detections and unnecessary blocks. Evaluate both kinds of error using representative legitimate traffic and adversarial cases. A filter that blocks medical terminology indiscriminately may make an otherwise permitted service unusable.
A second model can also check an answer for particular defects, but it needs its own evaluation. There is no general rule that multiplying two headline accuracy figures gives the reliability of a generator-verifier system. What matters includes the verifier’s ability to catch the generator’s errors, whether their mistakes are correlated, and what happens after a rejection.
For factual responses, checks might compare cited passages with claims, confirm values against a system of record, or restrict answers to supported material. A verifier’s approval is evidence from that checking process, not proof of truth. If sources are missing or contradictory, the application should have a defined route to abstain or request help.
Avoid unlimited retries. Set limits on attempts, cost, and elapsed time, and specify what happens when a valid result cannot be obtained. Otherwise a rejected answer can become a slow and expensive loop that still ends with an unsuitable output.
Approval Must Refer to a Specific Action
Human confirmation is useful only when the person can assess the relevant facts and understands what will happen. Show the recipient, amount, reason, and supporting records before a consequential transaction. Preserve a reject or escalation route, and allow enough time for the required review.
Bind approval to the reviewed proposal. If the amount or recipient changes afterwards, the earlier approval should not authorise the new transaction. Recheck relevant state at execution because another process may have already issued the refund. Use transaction identifiers and idempotency controls where retries could repeat an external action.
The interface ergonomics article examines the reviewer’s evidence and workload. Adding a person to an approval chain cannot compensate for an inaccessible source record or a queue that makes meaningful checking impractical.
Decide How the System Fails
If the policy service is unavailable, the assistant might save a draft and direct the case to a manual queue. It should not invent eligibility rules to keep the conversation moving. If a nonessential recommendation feature fails, the process may be able to continue without it. Failure behaviour should follow the consequences of the missing check.
Monitoring and logs support detection and investigation, but they are not substitutes for prevention. Record enough to connect a request, retrieved evidence, proposed action, approval, and actual outcome. Limit sensitive content, control access, and define retention rather than collecting everything indefinitely. Alerts also need owners and response procedures.
Exercise the stop mechanism and fallback path before depending on them. NIST’s AI RMF Core includes mechanisms and responsibilities for superseding, disengaging, or deactivating systems whose outcomes conflict with intended use. In practice, someone must be able to disable an unsafe capability while preserving pending cases and determining what has already happened.
Maintain Guardrails as the System Changes
Test controls together, including their failure modes. A new tool, model release, document source, or policy can open a path that earlier tests never exercised. Keep regression cases for incidents and near misses, and test that blocked actions remain blocked after changes. Include false-positive rates, latency, and fallback workload in acceptance criteria.
Assign ownership to the control and the underlying rule. A refund threshold may need a business policy update; an authorisation defect needs a technical correction; a misleading approval screen needs workflow design. Clear ownership lets the organisation respond to the actual failure. Effective guardrails are maintained limits on a working system, supported by evidence that those limits hold where they matter.
