A customer service team introduces AI to classify incoming requests and suggest which ones need urgent attention. During the pilot, experienced staff check its recommendations. Response times improve, and the team gradually routes more work through the system. Several months later, a product change produces an unfamiliar type of complaint. The classifier sends these requests to a low-priority queue, where they wait until customers start calling again.
The immediate problem is a classification error. The more difficult questions concern the surrounding process. Can staff find the affected requests? Who can change the routing rule? Can the team handle the queue while the classifier is investigated? A system that was introduced to support the service may now determine how the service operates, without anyone having explicitly approved that wider dependence.
This is one way control can weaken during AI adoption. It is a possibility to examine, rather than an inevitable outcome. The practical issue is whether the organisation retains enough evidence, authority, and capacity to change course as its reliance on the system grows.
Control Means Being Able to Intervene
Operational control involves understanding a system well enough to decide when and how to act. For the service team, that means reconstructing a routing decision, correcting it, changing future behaviour, and continuing essential work during an interruption. These capabilities depend on the model, application, supplier arrangements, and people operating the process.
Control does not require every employee to understand every model parameter. Different decisions require different evidence. A service manager investigating a delayed complaint needs to see the original request, the assigned category, the routing policy, and subsequent actions. An engineer investigating a regression needs configuration and version information. Neither task is satisfied by a dashboard showing average accuracy alone.
This operational focus gives the subject a different emphasis from accountability in algorithmic systems. Accountability concerns who must answer for outcomes and provide correction. Control concerns whether the organisation can actually carry out an intervention, including when its usual process is unavailable. The responsibilities and the practical capability need to meet.
Dependence Can Grow Beyond the Approved Use
The service team’s pilot may have authorised AI to suggest categories. Connecting those categories to automatic routing changes the consequences of an error. Reducing manual triage capacity changes them again. Each step can look reasonable on its own while the original evaluation becomes less representative of the operating arrangement.
A useful adoption record therefore describes the full workflow. It identifies which outputs remain suggestions, which trigger actions, which cases require review, and which decisions remain outside the system’s authority. When a category starts determining access to an urgent queue, the owner should reassess that use before treating it as a routine integration change.
Dependencies also develop outside the main application. Reporting may use the categories to allocate staff. Another team may use them to identify product defects. Changing the classifier can then affect decisions that its original owner does not know about. Sculley and colleagues’ research on technical debt in machine learning systems identifies related problems involving data dependencies, hidden feedback loops, and undeclared consumers of model outputs.
For this team, a dependency record should identify the queues, reports, and downstream services that consume each classification. It gives a proposed change an audience: the people who need to test the effects and prepare their part of the process.
Review Needs Evidence and Permission to Change the Outcome
Human review can preserve control when reviewers have relevant information, sufficient time, and authority to act. Adding an approval button provides little assurance if staff see only the suggested category and are expected to clear hundreds of requests an hour.
The reviewer should be able to inspect the request, recognise uncertainty, redirect it, and escalate a recurring problem. The interface should make the consequences clear. Correcting one category might move a request without changing the classifier. Adjusting a routing rule might affect every future request in that category. Retraining a model is a separate intervention with its own evaluation needs.
These distinctions make override records useful. Record what staff changed and why, then examine representative cases alongside outcomes. A low override rate might reflect reliable recommendations, missing information, time pressure, or reluctance to challenge the system. A high rate might reveal poor performance or an intentionally cautious review policy. Neither rate establishes the quality of oversight by itself.
Explanations need similar scrutiny. A generated account of why a request was classified is not necessarily a faithful record of the factors that produced the classification. Where evidence is limited, the process should acknowledge that limitation and provide another route to review. A confident explanation should never substitute for the original request and the actions actually taken.
Legal requirements also depend on context. In US credit decisions, for example, Regulation B, section 1002.9(b)(2) requires specific principal reasons in a statement of reasons for adverse action; failing to reach an internal qualifying score is insufficient. That requirement should not be generalised into a universal duty to explain every AI output, but it shows why evidence requirements must be established for the particular use.
Monitoring Should Follow the Consequences
For the service classifier, overall accuracy is useful but incomplete. A small class of urgent complaints could be consistently misrouted while common categories remain accurate. Evaluation should examine the errors that matter to the service, including urgent requests assigned low priority, delays before correction, and differences across relevant request types or languages.
The comparison needs a credible reference. Agreement between a model and its own generated labels does not establish correctness. Staff may need to review a sample independently, using agreed criteria, and reconcile disagreements. Where outcomes take time to emerge, monitoring should distinguish early indicators from confirmed results.
Changes in incoming data deserve investigation, but a changed distribution does not automatically mean performance has worsened. Conversely, performance can deteriorate without an obvious shift in simple input statistics. A new product might generate familiar language with a different operational meaning. Monitoring should connect data checks with sampled decisions and downstream service outcomes.
Feedback can complicate the evidence further. If staff inspect only requests already marked urgent, those requests generate more correction data than the low-priority queue. Training on those corrections alone may leave missed urgent cases underrepresented. Sampling across queues helps the team examine what its current workflow otherwise leaves unseen.
Monitoring becomes a control when it leads to a defined response. An alert should identify who investigates, what evidence they need, and which temporary measures they can authorise. Otherwise the organisation has visibility without a reliable way to act on it.
Behavioural Changes Need a Release Record
An AI application’s behaviour can change through a new model, prompt, retrieval source, routing threshold, or data transformation. A supplier may also change a hosted service. Application code history alone may therefore be insufficient to reconstruct what happened.
The release record should identify the components and settings relevant to the decision, along with the evaluation evidence and approval. Logging should support investigation while respecting privacy, retention requirements, and access restrictions. Retaining all customer content indefinitely would create a separate problem; retaining too little may make a consequential decision impossible to investigate.
Before changing the classifier, test examples drawn from the actual service, including urgent cases previously missed. Compare the proposed version with the existing arrangement and examine material regressions. A staged release can limit exposure while staff check whether the expected improvements appear in use.
Rollback also needs qualification. Restoring an earlier model does not automatically repair requests already misrouted. Recovery may require identifying affected records, reprioritising them, and contacting customers. If the incoming requests have changed, the earlier model may no longer be suitable either. The recovery plan needs to cover the service backlog and current conditions as well as the software.
Fallback Capacity Has to Survive Adoption
A manual process can look reassuring in documentation while being impractical at production volume. If the service team has removed most triage capacity, switching off the classifier may create a queue that grows faster than staff can review it. The organisation still has a stop button, but using it now carries substantial consequences.
Fallback planning should specify which work continues, at what capacity, and for how long. The team might prioritise safety-related complaints manually, pause non-essential categorisation, and communicate longer response times. Managers need to agree these choices while there is time to prepare them.
The NIST AI Risk Management Framework’s Manage function explicitly addresses responsibilities and mechanisms to supersede, disengage, or deactivate systems whose outcomes are inconsistent with intended use. In this service, an exercise could test whether staff can identify urgent work and maintain an agreed response level during an outage.
Supplier arrangements belong in that exercise. Check whether records and configurations can be exported, what notice is available for material changes, and whether a previous model version remains accessible. An alternative supplier is useful only if the organisation can integrate and evaluate its service within the required recovery period.
Make Growing Dependence a Deliberate Decision
Retaining control takes effort, but the amount should reflect the consequences of the workflow. An assistant suggesting internal labels may need modest checks. A classifier determining which customers receive urgent attention needs stronger evaluation, intervention authority, and recovery arrangements. Model complexity alone does not settle that choice, and simpler systems can also become difficult to change through poorly understood dependencies.
The service team can review control as adoption expands: demonstrate a correction, trace a recent decision, test a release change, and exercise reduced-capacity operation. These activities reveal whether the approved process still matches the one people rely on.
AI can improve a service while increasing its dependence on particular systems and suppliers. The organisation needs to decide how much dependence it can support and maintain the capabilities that make intervention possible. That decision becomes more credible when the team can show how it would recognise a problem, change the process, and continue serving people while the change takes effect.
