A product team uses a generative AI application to prepare answers to customer questions. The answers are clear, concise, and usually plausible. The difficult cases concern what the text commits the business to: a feature that does not exist, a refund exception that has not been approved, or a delivery date that the source material never promised.
Governance for generative AI needs to examine how generated content becomes consequential. Drafting, searching, summarising, publishing, and acting are different uses, even when they share the same underlying model. A suitable control for a private brainstorming session may be inadequate when an answer is sent to a customer or stored as an authoritative record.
The application also includes more than the model. Prompts, source documents, retrieval, interfaces, tools, user expectations, and review processes all influence the result. Governing the deployment means establishing appropriate conditions across that complete workflow.
Connect Generative AI Use to Business Intent
Generative AI governance combines policy, decision rights, technical safeguards, and monitoring for systems that produce text, images, audio, video, or code. Its purpose is to enable useful applications within an agreed risk appetite, including clear limits on what generated material may represent or authorise.
Give each initiative a short charter: the business objective, intended audience, permitted content, unacceptable outcomes, accountable owner, and approval route. Register the application and its model services, including embedded features and unapproved use discovered through staff feedback or software reviews. This helps the organisation identify shadow AI and decide which uses to support, restrict, or replace with approved services.
For the product team, the charter might permit staff-assisted replies supported by current documentation, with contractual exceptions reserved for an authorised colleague. The business benefit and the control boundary then refer to the same workflow. The operating-model article explains who should maintain that boundary and decide on exceptions.
Define What the Output Is Allowed to Become
Describe the intended use in terms of the work people will perform. An application might prepare suggested responses that staff check against current product information. That approval does not automatically cover answering customers without review or updating contractual records.
Set boundaries around subject matter, audience, data, and actions. For example, the application could answer questions about documented product features while routing pricing exceptions, contractual interpretation, and complaints to the relevant team. Make those boundaries visible in the interface and operating guidance.
Consider how users may expand the task. A successful summarisation tool can gradually become a source of advice because its explanations sound useful. Monitoring and user feedback should help identify that expansion before it becomes an unexamined dependency.
Assign an owner who understands the business consequences of the output. Technical specialists can evaluate model behaviour, but they need process expertise to establish what counts as an unacceptable commitment or an important omission.
The business governance framework connects this scope to assessment, approval, and monitoring. Generative AI adds specific questions about how variable content is checked and what evidence users receive.
Evaluate Plausibility and Factual Support Separately
Language quality does not establish that a statement is supported. An answer can be grammatically sound and consistent in tone while making a claim absent from the evidence. It can also summarise individual sentences accurately while omitting a qualification that changes their practical meaning.
NIST’s Generative AI Profile identifies confabulation among the risks of generative AI. That provides a reason to evaluate factual support explicitly, rather than assuming fluent output indicates dependable understanding.
For the product application, distinguish unsupported feature claims, incorrect dates, missing conditions, and harmless stylistic variation. These errors have different consequences and should not disappear into one overall quality score.
Evaluate the answer against an authoritative source and the actual question. A response can quote a genuine document that does not apply to the customer’s product version. Review therefore needs to examine relevance, currency, and the connection between evidence and claim.
Generated citations need checking too. A reference may exist without supporting the sentence beside it, or an application may produce a plausible-looking reference that cannot be located. Where citations are part of the control, test their validity and usefulness for the reviewer.
Consider an answer about delivery that combines a standard dispatch period with a separate document describing an optional express service. Both sources may be genuine, yet the combined answer could promise express delivery to a customer whose order does not qualify. An evaluation that checks only whether the cited pages exist would miss the error. The relevant test examines whether the conclusion follows from the sources under the customer’s conditions.
Manage Retrieval as an Information Service
Retrieval-augmented generation supplies selected source material to the model at request time. It can improve access to relevant information, but it does not establish that the sources are correct or that the generated answer faithfully represents them.
Give source material an owner. Record which product guidance is approved, when it takes effect, and how obsolete versions are removed or clearly distinguished. The application needs a defined response when sources conflict or do not answer the question.
Preserve access boundaries. A customer-facing application should not retrieve confidential product plans simply because they resemble the user’s question. Check permissions before the material is included in the model request, and examine caches, conversation sharing, and logs for additional routes to disclosure.
Monitor retrieval separately from generation. If an answer is wrong, establish whether the application found the wrong document, missed an important passage, or generated an unsupported statement despite receiving suitable evidence. These diagnoses lead to different corrections.
Source changes need verification as well. When a product feature is withdrawn, check whether the old guidance remains in the index, whether cached answers still mention it, and how existing conversations handle the update. A document-management change is complete only when the application uses the intended information under the conditions that matter for the service.
The privacy-risk article follows personal information through these stages. A model-training restriction cannot substitute for appropriate control over retrieval and the records the application creates.
Establish Data Rights and Supplier Evidence
Record the origin, permitted use, and restrictions of material used for training, fine-tuning, retrieval, and evaluation. Ask suppliers what they can substantiate about training-data provenance, licensing, privacy controls, retention, and subcontractors. Document gaps in that evidence rather than assuming access to a model establishes rights to every input or output.
Assess the proposed publication separately from the permission to process its source material. A product image, recorded voice, or generated quotation can raise different rights and disclosure questions from an internal text summary. Check relevant privacy, intellectual property, and regional AI obligations for the actual use. The compliance article provides the broader legal assessment process.
Test Representative Work and Important Exceptions
Build an evaluation set from the proposed use, including straightforward questions, ambiguous requests, missing information, conflicting guidance, and cases outside scope. Include the languages, document formats, and user circumstances that matter for the deployment.
Define acceptable behaviour before scoring outputs. A good answer may sometimes be a limited explanation, a request for clarification, or referral to an authorised person. Penalising every non-answer can encourage an application to produce confident responses where the evidence is insufficient.
Keep evaluation evidence separate from development examples where possible. If prompts are repeatedly adjusted to satisfy a familiar set, that set becomes less informative about performance on unseen work. Use additional cases and human review appropriate to the consequences of the release.
Repeated trials can help reveal variability on important inputs. They do not need to produce identical wording, but material claims and boundaries should remain acceptable. Model settings, version information, prompts, source snapshots, and evaluation criteria should be recorded so results can be interpreted.
Automated grading can help sort large volumes, but check whether it detects the errors that matter. A model assessing another model’s answer may reward plausible wording or miss the same unsupported assumption. Use domain judgement and direct checks against source evidence where those are necessary.
Evaluate the user workflow too. An answer that is technically reviewable may still be impractical to check if its sources are hidden or the user has little time. Measure correction and review effort alongside accepted output, particularly when the investment case depends on saving staff time.
Place Review and Security Controls Where They Can Work
Human review needs a defined purpose. A reviewer might verify factual claims, approve external commitments, or assess whether an answer fits a customer’s circumstances. Provide the information and authority needed for that decision, and clarify what requires escalation.
For product answers, show the applicable source passage, relevant product version, and any missing information. Make it easy to edit or reject a suggestion. If the organisation expects staff to check every claim, include that effort in its staffing and service expectations.
Security controls must also account for untrusted content. A retrieved document, user message, or attachment can contain instructions intended to redirect the application. The OWASP prompt-injection prevention guidance discusses this class of attack and layered mitigation. Treat source material as content to evaluate, with permissions and authorisation enforced outside the model.
Where the application has tools, restrict the functions, identities, and destinations available to it. A product-answering assistant may need to search approved documents without having permission to change prices or send messages independently. The agent governance framework examines those action boundaries in more detail.
Outputs entering other software need appropriate validation. Generated markup, queries, or structured fields should be treated as untrusted input to the receiving system. A text-generation feature can create ordinary application-security problems if its output is executed or rendered without suitable checks.
Include adversarial testing or red-team exercises proportionate to the deployment. Test attempts to override instructions, extract confidential material, manufacture sources, and generate harmful or misleading content. Record the failure, control response, and correction, then rerun relevant cases before expanding use. Assign a named business owner and technical operator who can restrict the feature; reviewers of critical outputs also need explicit override and escalation authority.
Make Content Provenance and Disclosure Useful
Keep enough provenance to identify the source material, model or service version, relevant prompts and settings, and review decisions behind a consequential output. Describe the application’s processing and limitations at a level users can understand. A model’s generated explanation should not be treated as a verified account of its internal reasoning.
For published synthetic media, consider visible labels and supported metadata or watermarking mechanisms, alongside any applicable legal disclosure requirements. Decide who checks that disclosures survive export and publication. Provenance can help establish origin without proving that a claim is true, and a missing watermark cannot establish that content was created by a person.
Define which drafts may remain informal and which outputs need an audit record before external use or entry into a business system. For a customer answer, that may mean retaining the applicable source version and approval rather than every exploratory conversation. Human review should be strongest where an unsupported claim could create a significant commitment or affect a person’s interests.
Reassess the Service as Its Dependencies Change
A supplier can alter a model or service behaviour without the business changing its application code. Source documents, prompts, retrieval settings, and user behaviour can also change the conditions that earlier tests examined.
Maintain an identifiable configuration and a change process. Before a material release, run relevant evaluations and investigate meaningful differences. Where versions can be pinned, understand what that does and does not preserve, including service dependencies outside the selected model version.
Monitor a proportionate sample of production use, complaints, corrections, unsupported claims, and out-of-scope requests. Protect personal and confidential information in that monitoring. Broadly retaining every conversation for possible analysis can create additional obligations and exposure.
Agree when to restrict or suspend the feature and how users continue their work. A fallback might return staff to approved product guidance while a problematic capability is investigated. A rollback should include affected prompts, sources, or integrations where necessary, rather than assuming the model alone caused the issue.
Use Generative AI Guidance to Check the Design
NIST’s AI RMF organises risk work around Govern, Map, Measure, and Manage. Its Generative AI Profile, NIST AI 600-1, adds generative-AI-specific considerations. ISO/IEC 42001 addresses the management system supporting such work, rather than certifying that individual generated answers are accurate.
For regional implementation examples, New Zealand’s Responsible AI Guidance for the Public Service: GenAI addresses public-service use. Singapore’s Model AI Governance Framework for Generative AI covers areas including accountability, data, testing, security, incident reporting, and content provenance. Use these references to challenge the design while establishing applicable legal duties separately.
The governance operating model assigns responsibility for these decisions. For the product team, dependable governance means the application has a bounded role, its claims can be checked against current evidence, and changes in performance reach someone able to act before unsupported answers become routine business commitments.
