A delivery platform builds a model to predict which orders are likely to arrive late. Operations uses the score to prioritise intervention. Staff contact drivers, adjust routes, and notify customers before problems escalate. After deployment, some high-risk orders arrive on time because those interventions worked.
The next training dataset now contains outcomes influenced by the previous model. If the team evaluates prediction quality without recording interventions, it may interpret successful prevention as a forecasting error. If it retrains without examining that relationship, it may weaken a useful warning signal.
This is one reason algorithmic warnings deserve more than a place on a presentation slide. They describe different mechanisms: a target can distort behaviour, a reward can admit shortcuts, and a prediction can change the outcome being predicted. Distinguishing those mechanisms helps a team choose tests that address the actual problem.
What the Familiar Warning About Targets Actually Says
“When a measure becomes a target, it ceases to be a good measure.” This wording appears in Marilyn Strathern’s 1997 paper on audit in British universities, where she discusses Goodhart’s law. It should not be presented as a verbatim quotation from Goodhart’s original monetary-policy paper.
The warning concerns the relationship between an indicator and the activity it represents. A measure that was informative under one set of incentives may become less informative when improving it carries rewards or penalties. That possibility is a reason to inspect the relationship, rather than proof that every target necessarily becomes useless.
For delivery operations, an on-time rate may be useful alongside other service measures. If staff can improve the reported rate by changing promised times after an order is placed, the number no longer has the same meaning. If they improve it by resolving route constraints, the measure may reflect a real benefit. The result depends on what changed and how the metric is defined.
The first diagnostic step is therefore concrete: identify the easiest ways to improve the indicator without improving the intended outcome. Then check whether the process permits them and whether the data would reveal them. The practical guide to metric review develops that organisational work in more detail.
Specification Gaming Is About the Objective That Was Implemented
Specification gaming occurs when behaviour satisfies the implemented objective while departing from the intended task. It is particularly visible in reinforcement learning, where an agent can discover a way to collect reward that its designers did not intend.
DeepMind’s discussion of specification gaming includes a block-stacking example: an agent could satisfy an incomplete height-based requirement by flipping a block rather than stacking it as intended. The example illustrates a mismatch in the specification; it is not evidence that an agent had a human-like intention to deceive.
Business systems can have analogous mismatches without using reinforcement learning. A route optimiser may minimise distance while omitting a contractual delivery window. A scheduling tool may reduce overtime while producing impractical shift transitions. These are possibilities to test against the actual implementation, rather than inevitable effects of optimisation.
Constraints should be expressed where they can be enforced. If a route must respect a vehicle restriction, validate that requirement in the routing system. A favourable average score should not compensate for a violation that the business considers unacceptable. Reviewers need to understand which requirements are hard constraints and which are preferences that may be traded off.
Predictions Can Change Their Own Operating Environment
The delivery example involves a different mechanism. A risk prediction triggers action, and that action changes arrival times. Perdomo and colleagues’ research on performative prediction formalises settings in which predictions influence the distribution of outcomes. This can create difficulties for evaluation and retraining when the relationship is ignored.
The feedback need not be harmful. Warning staff about a late delivery is useful precisely because it can lead to prevention. The problem is treating the observed arrival as if no intervention occurred. A model that accurately identifies preventable delays can appear overcautious after staff resolve them.
Record which actions followed a prediction, when they occurred, and what resources were used. Evaluate the operational process as well as the predictor. Relevant outcomes might include avoided delays, intervention workload, and whether customers received useful notice. Prediction accuracy alone cannot establish the value of the intervention policy.
Estimating what would have happened without intervention requires care. Where appropriate, a controlled comparison or phased rollout may help. In other settings, withholding assistance would be unsuitable. The team should document what its evaluation can establish and seek suitable statistical and operational advice for consequential comparisons.
Selection Can Hide the Cases Most in Need of Investigation
A system may determine which cases generate detailed evidence. Fraud flags receive investigation; unflagged transactions may receive less scrutiny. A recommendation receives exposure and can attract clicks, while an unseen item has no comparable opportunity. The resulting dataset reflects selection as well as the behaviour the model is intended to assess.
Suppose the delivery team records detailed failure reasons only for orders the model marked high-risk. A new type of delay occurring among low-risk orders may then be poorly represented in the next training cycle. This is a hypothetical design risk that an evaluation should investigate, not an assertion that every monitoring process has the defect.
Sampling across the full workflow helps reveal what the usual review path leaves unseen. Include cases predicted to be routine, cases where staff intervened, and cases where they did not. Keep selection rules visible so the analyst knows how each label was obtained.
More data from the same selective process may preserve the blind spot. The team needs evidence that reaches the missing cases, rather than an assumption that scale will eventually make the dataset representative.
Strategic Adaptation Requires Its Own Tests
Once people know that a score affects an outcome, some may adapt their behaviour. Adaptation can be legitimate: a supplier improves delivery practices after learning what reliability measures capture. It can also be superficial or adversarial: records are altered to appear compliant while the underlying service is unchanged.
The distinction matters because not every response to a metric is gaming. A useful system is often intended to encourage better behaviour. The question is whether the change improves the intended outcome or only its recorded appearance.
Threat modelling should identify who can influence the inputs, what they gain, and which manipulations are feasible. Test concrete cases such as duplicated events, delayed timestamps, or selective reporting where those are relevant. Keep the scope proportionate to the incentives and the consequences.
Changes after deployment also need investigation before attribution. A sudden improvement might reflect better operations, a seasonal shift, a changed definition, a software defect, or manipulation. An alert should initiate a check of those explanations, rather than automatically trigger retraining or accusations of bad faith.
Monitor the Mechanism and the Operational Consequence
A monitoring plan should connect each suspected failure mechanism to evidence and action. For specification errors, inspect constraint violations. For target distortion, compare the indicator with independent outcome evidence. For feedback, retain intervention records. For selective labels, examine sampling coverage. For adversarial adaptation, investigate feasible input manipulation.
These checks complement ordinary service monitoring. The application still needs reliable data pipelines, acceptable latency, access controls, and incident handling. A failed import can resemble changed customer behaviour, so teams should rule out instrumentation problems before explaining every shift as a model phenomenon.
Version the settings that affect decisions, including thresholds and preprocessing. When behaviour changes, the team needs to know whether the cause could be a release. Keep a route to restrict the affected workflow and reassess outstanding cases, with responsibilities agreed before an incident.
Use the Warning to Design a Specific Evaluation
For the delivery platform, an evaluation can pair a delay forecast with a record of the intervention it triggered. The report should distinguish orders with no intervention, orders where staff acted, and orders where an action was proposed but could not be completed. Differences among those groups may reveal operational constraints as well as modelling issues. They do not, without further analysis, establish what caused a better outcome.
A separate test can examine whether the routing recommendation respects delivery windows and vehicle restrictions. That test assesses a specification requirement and need not rely on the same metric as forecast evaluation. An input audit can then check timestamp validity and whether all relevant orders enter the reporting pipeline.
Keeping these questions separate makes corrective work clearer. A violated route constraint calls for a change to validation or optimisation. Missing intervention records call for better instrumentation. A weak forecast calls for investigation of the predictor and its inputs. Replacing the model in every case would leave some causes untouched, even if the next benchmark result improved.
The guide to algorithmic consequences examines the people affected when those controls fail. This technical assessment asks how a problem could arise and what evidence would expose it. Together, they support an evaluation that looks beyond a successful benchmark and follows the system into use.
Algorithmic warnings become useful when they help a team anticipate a particular failure and design a way to detect it. Targets, reward specifications, interventions, and data selection can each change what a reported result means. Examining those connections gives the organisation a better basis for improving the system while preserving the purpose it was introduced to serve.
