A procurement reviewer opens a supplier assessment. The first screen recommends Supplier A, gives it a score of 82, and explains why it offers the strongest proposal. The reviewer then opens the bids, checks several details, and approves the recommendation. The decision record now contains both an automated assessment and a human approval.
That record does not show whether the reviewer would have reached the same conclusion before seeing the recommendation. It also does not establish that the approval was biased. The reviewer may have checked the evidence carefully and found the proposal sound. What it cannot supply is evidence of a separately formed initial judgment, because the workflow never collected one.
This distinction matters when an organisation presents human approval as independent confirmation. A person can be responsible for a decision while having formed it through interaction with the system’s answer. The design needs to match the kind of assurance the organisation claims.
Independence has more than one meaning
A reviewer may be organisationally independent of the model developer, authorised to override the output, and still influenced by what the interface presents first. Organisational separation, decision authority, and independence of assessment are different properties.
An AI recommendation can establish a starting point for comparison. Its score may become the number the reviewer adjusts, while its explanation identifies the questions the reviewer considers. Details outside that explanation may receive less attention. These are plausible routes by which the initial answer can shape the task; their strength depends on the people, interface, evidence, and decision involved.
That influence is not always harmful. A useful recommendation may direct attention to a requirement the reviewer missed. The aim is appropriate reliance: using correct assistance while retaining the ability to detect and reject incorrect assistance. Treating every change of mind as failure would discard part of the system’s intended value.
NASA’s study of automation bias in cockpit decisions examined the substitution of automated cues for further checking. Its setting differs from procurement, but it provides a reason to investigate how people seek evidence after receiving advice rather than assume that an approval represents a fresh assessment.
The recommendation can define what gets checked
Suppose Supplier A’s proposal contains a condition that conflicts with the organisation’s data-location requirement. The generated assessment does not mention it. A reviewer checking the explanation sentence by sentence might verify every stated claim and still fail to examine the omitted condition.
The problem is broader than accepting an incorrect sentence. The system has selected a frame for review: these are the strengths, these are the concerns, and this is the preferred outcome. If the reviewer uses that frame as the checklist, an omission can survive without any explicit falsehood being endorsed.
Scores add another interpretive question. A displayed 82 could represent a defined calculation, a model-generated judgment, or a combination of rules and estimates. The reviewer needs to know which it is and which evidence supports the components. Numerical precision alone does not establish calibration or agreement with the procurement criteria.
An explanation can help when it exposes relevant evidence and assumptions. It can hinder a check when its fluent presentation becomes a substitute for locating them. The practical distinction is whether reviewers can test the account against the bids and criteria, including points the account never raises.
Form an initial assessment where it adds useful protection
One design option is to ask the reviewer for a preliminary assessment before showing the automated recommendation. For procurement, that might mean recording whether each bid meets a few decisive requirements and noting unresolved questions. It need not mean reproducing a full generated report by hand.
Buçinca, Malaya, and Gajos tested cognitive forcing designs in an experiment with 199 participants. The designs reduced overreliance relative to simpler explanation-based approaches, with less favourable subjective ratings for the most effective designs. This supports testing the sequence and effort of interaction; it does not prove that asking people to go first will improve every professional decision.
An initial assessment has costs. It requires time and sufficient expertise, and it may itself become an anchor that makes useful advice harder to accept. Choose the task and level of detail deliberately. A brief independent check of a critical requirement may offer more value than requiring a second complete report that nobody has capacity to examine.
Sequence also does not create independent information. If both parties use an incomplete extract of the bids, recording the review first cannot restore the missing clause. The redundancy paradox of human oversight examines that separate problem of shared failure sources.
Use disagreement to find evidence, not to assign a winner
A difference between the initial assessment and the recommendation identifies something to investigate. It does not establish that the model is wrong or that the reviewer should prevail. Compare the source evidence, the applicable criteria, and any assumptions that produced the difference.
For the conflicting contract condition, the review should identify the exact clause, determine whether it applies to the proposed service, and establish who can resolve an ambiguity. The outcome might be a corrected assessment, a request for clarification, or a decision that the bid cannot yet be ranked confidently.
Keep the preliminary and final records distinguishable. A later investigator should be able to see what changed and why. Updating the initial assessment to match the final answer would remove the evidence the sequence was designed to preserve.
Do not turn disagreement into a performance target. A reviewer who invents objections to demonstrate independence contributes little assurance. A reviewer who correctly agrees with accurate advice may be doing the job well. Performance metrics and judgment need to be considered together when setting review expectations.
Measure whether advice improves the final decision
Agreement rates alone cannot answer whether the assistance is useful. Evaluate decisions against a separately established reference where the task permits one, and retain uncertainty where the evidence supports several defensible outcomes.
For cases with an adjudicated answer, useful patterns include:
| Initial assessment and AI advice | Final result to investigate |
|---|---|
| Reviewer wrong, AI correct | Did the advice enable a justified correction? |
| Reviewer correct, AI wrong | Did exposure cause a harmful switch? |
| Both wrong | Did either stage seek evidence that could reveal the error? |
| Both correct | Did the process preserve correctness at a reasonable cost? |
This framing recognises both the benefit of assistance and the risk of overreliance. It also distinguishes an accurate decision from a superficially independent process. Two separate assessments can agree and still be wrong; an assessment influenced by good advice can become more accurate.
A controlled evaluation could compare recommendation-first review with a preliminary-assessment design using appropriately allocated, representative cases. Include correct and incorrect recommendations. Balance case difficulty and reviewer experience, and account for learning or memory if people see related material more than once.
Measure time, unresolved cases, harmful switches, useful corrections, and the final decision’s quality. An intervention that reduces acceptance of bad advice but prevents more beneficial corrections needs closer examination. A single overall accuracy measure can conceal that trade-off if the evaluation set contains very few erroneous recommendations.
If the interface displays confidence, evaluate what that display changes separately from the recommendation’s content. Compare otherwise similar cases and record whether reviewers seek evidence, revise a correct initial assessment, or appropriately retain uncertainty. A higher reported feeling of confidence after review does not establish a better decision. Keep the quality reference independent of the persuasive features being tested, and avoid using the system’s own score as the answer against which the reviewer is graded.
Keep the safeguard usable in ordinary work
A preliminary assessment only helps if it contains real evaluation. If reviewers must clear a large queue quickly, a required text box may produce generic statements rather than additional checking. The design should make the decisive evidence accessible and give uncertainty a usable route through the process.
The interface can separate source documents, extracted facts, model inferences, and the final recommendation. Showing an extracted clause with a source location lets the reviewer verify it. Showing only another generated summary leaves the reviewer dependent on an intermediate interpretation even when the preferred supplier remains hidden.
Record missing evidence explicitly. If a bid does not establish a required condition, reviewers should be able to hold the assessment for clarification instead of guessing in order to submit a form. The process needs an owner for that unresolved work and a way to prevent premature commitment.
The effort question belongs to the economics of automated judgment. Sequence changes should be evaluated alongside workload and escalation capacity, rather than introduced as an extra ritual on top of an unchanged daily target.
Independent judgment is a property to design and examine, not something established by a person’s name on an approval record. Be clear whether the review is intended to provide a fresh assessment, a targeted source check, or an accountable decision informed by advice. Each can be useful when the workflow supplies the evidence and conditions that its purpose requires.
