A maintenance team uses an AI assistant to summarise equipment reports and prepare proposed work orders. Staff review the proposals before they enter the scheduling system. The workflow appears reliable until a document-processing change drops a qualification from one report. The assistant proposes a routine inspection where the source called for specialist assessment.
The immediate error is visible in the proposed order, but investigating it requires several kinds of evidence. Which report version was processed? What text reached the model? Did the reviewer see the original qualification? Has the same processing change affected other proposals, and can the team stop them from reaching scheduling while it investigates?
Production safety connects risk decisions to those operating capabilities. It includes the model, surrounding software, data, users, permissions, and recovery process. Its purpose is to reduce the likelihood and consequences of unsafe outcomes while giving the organisation a workable response when its assumptions prove incomplete.
Start With Hazards and the System’s Permitted Role
Define the intended task and the consequences of failure before choosing controls. For the maintenance assistant, relevant hazards could include omitted warnings, incorrect equipment identification, unsupported recommendations, and delays in specialist review. The assessment should identify who could be affected and which parts of the existing maintenance process remain authoritative.
The assistant might prepare a draft work order while the scheduling system enforces required approvals. Permission to prepare text should not imply permission to authorise maintenance or change equipment settings. If the organisation proposes wider authority later, that change needs a fresh assessment of consequences and evidence.
This work belongs within governance. The NIST AI Risk Management Framework explicitly treats risk management as continuous across the lifecycle and includes monitoring, incident response, recovery, and mechanisms to supersede or deactivate unsuitable systems. It should not be characterised as a one-time model approval that ignores production.
The implementation question is whether those expectations have become working controls. A policy requiring specialist review needs a defined trigger, an available reviewer, and an enforced path that cannot silently treat a missing approval as permission to proceed.
Evaluate the Complete Workflow Before Expanding Authority
Testing should cover the path from source report to reviewed proposal and downstream action. Include incomplete documents, conflicting identifiers, unusual equipment histories, unsupported requests, and failures in dependent services. These cases test assumptions that an evaluation of the language model alone may miss.
The maintenance team needs agreed expectations for each important case. A proposal that cannot be resolved from the available evidence should remain unresolved. Filling a missing equipment identifier with a plausible value may produce a valid-looking order that refers to the wrong asset.
Reviewers should test whether the interface makes such problems detectable. Can staff compare a proposal with the source? Are unresolved fields visible? Does rejecting a proposal preserve the original request for further work? Measuring review speed without checking error detection would leave an important part of the control unexamined.
Begin with bounded exposure where the evidence justifies proceeding. A shadow stage can generate proposals without changing the scheduling system, followed by limited supervised use. Record which conditions remain untested and which must be resolved before the assistant can handle greater volume or more consequential actions.
Make Behavioural Changes Visible and Reviewable
Behaviour can change through model updates, prompts, retrieval sources, document parsers, preprocessing, thresholds, and application rules. A stable application interface does not establish that the decision process is unchanged. Keep a release record covering the components that materially affect outputs and actions.
Distinguish a release from environmental change. A fixed model can become less suitable when incoming reports or operating conditions change. Retraining changes the model and should be recorded as a release. Neither event should be hidden under a general label such as model decay when a more specific account is available.
For the document-processing change, compare extracted content and resulting proposals against representative reports before deployment. Evaluate known failure cases and important regressions, including effects on small but consequential categories. A better average result does not automatically justify losing a critical qualification in specialist cases.
Supplier changes need equivalent attention. Establish what version information and change notice are available, what can be tested in advance, and which aspects the organisation controls. Where exact reproduction is unavailable, preserve the actual output used and document the limits of later reconstruction.
Monitor Outcome Quality and the Data Needed to Assess It
Availability and latency monitoring remain necessary, but a healthy service can still produce unsuitable proposals. Monitor failures that matter to the workflow, such as missing identifiers, unsupported recommendations, reviewer corrections, unresolved queues, and downstream rework. Define who examines each signal and what action they can take.
Changes in input distributions can prompt investigation without proving performance deterioration. Equally, a simple distribution check may miss a new meaning attached to familiar wording. Combine data checks with sampled decisions and operational outcomes instead of treating drift detection as a complete safety assessment.
Sampling needs to cover more than low-confidence cases. Reviewing only those cases provides a selected view and can miss confident errors. It does not, by itself, make a fixed model learn to express greater confidence. Such a change would require an update mechanism or optimisation pressure that actually influences its behaviour.
If feedback enters future training, record how cases were selected and labelled. A dataset built only from difficult reviewed proposals may be unsuitable for estimating performance across all work. Monitoring and retraining need to account for what the collection process leaves unseen.
Keep Permissions and Intervention Authority Practical
Security controls should follow the actions available to the assistant. OWASP’s guidance on excessive agency identifies risks associated with excessive functionality, permissions, and autonomy. For this workflow, the assistant can be limited to retrieving authorised reports and preparing proposals, with approval and writes checked by the relevant application services.
External reports and attachments are input material, not instructions granting authority. Their content should not enable the assistant to bypass approval requirements or access unrelated records. Controls enforced at tool boundaries reduce dependence on the model correctly interpreting every instruction it encounters.
Operational authority also needs limits. Specify who can pause proposal generation, block submission, or invoke an alternative process. Some actions may be available immediately to an on-call responder; others may require a specialist decision. The responder should not have to discover those boundaries during an incident.
Test that the intervention works with real permissions and dependencies. A stop control that disables the user interface while a background worker keeps submitting proposals has not contained the workflow. Check queued and in-flight work as well as new requests.
Design Investigation Records Around the Decision Path
The team should be able to connect a report to the extracted content, model output, validation results, reviewer changes, and final scheduled order. A correlation identifier across services helps join that evidence. Retain the relevant versions and configuration references rather than rely on timestamps alone.
Not every incident requires a full explanation of model internals. Operations may first need to establish that a parser removed text or that a scheduling rule misread a field. Where model behaviour needs investigation, an appropriate explanation method can contribute evidence. The explainable AI guide describes what those methods can establish.
Logging should be proportionate to investigation needs and information sensitivity. Restrict access and retention, and avoid copying entire confidential reports into several unrestricted services. Source references, controlled evidence stores, and suitable redaction may support investigation with less duplication.
Ownership of the complete workflow remains important even where responsibilities are distributed. A parser team, model provider, and scheduling team may each need to investigate. A service owner should coordinate their findings and the operational response so the maintenance team is not left to resolve the handoffs itself.
Recover the Work as Well as the Software
An incident procedure should cover detection, containment, investigation, communication, and recovery. For the missing qualification, containment might stop submission of affected proposals while preserving them for review. Investigation should identify which source formats and processing versions were involved and whether accepted orders need reassessment.
Restoring a previous component may prevent further errors while leaving earlier actions unresolved. Determine who checks affected orders, obtains specialist advice, and corrects scheduling or communications. An explanation of the cause is useful, but recovery also requires capacity to carry out those actions.
Fallback should reflect the workload. Manual processing may support the original pilot volume but be insufficient after expansion. Establish which requests receive priority, how the queue is tracked, and what service level remains achievable. Exercise the process before relying on it during a real interruption.
The exercise should include a clear route back to normal operation. Check who confirms that the defect is corrected, which evaluation must pass, and how held work will be released without duplicating orders or overwhelming reviewers. Preserve the incident evidence before changing the affected components. A successful restart should include confirmation that pending work has been reconciled, rather than rely only on the service becoming available again.
Production safety improves when operational evidence feeds back into risk decisions. A recurring defect may justify a narrower role, stronger review, a different component, or suspension until the issue is resolved. Retaining control during adoption depends on preserving those options as reliance grows. The organisation should be able to demonstrate what the assistant may do, how problems become visible, and how people can intervene and recover the work.
