A model flags a supplier invoice for investigation. Its explanation highlights the invoice amount and a recently changed bank account. A reviewer sees two plausible reasons for concern. Before accepting the explanation, they need to establish what it actually claims: did those inputs change the model’s score, did a payment policy trigger the hold, or did a language model simply produce a convincing account after the result was known?
These are different kinds of explanation. Each can provide useful information, but each needs a different test. Investigating an AI decision begins by separating claims about recorded events, model behaviour, real-world causes, and business justification. That separation makes it possible to use explanation methods critically without dismissing all explanations as fiction.
State the Question Being Investigated
“Why was this invoice held?” can refer to several questions. An operational question asks which components ran and what rule created the hold. A predictive question asks how recorded inputs contributed to a score. A causal question asks whether changing a real-world condition would change an outcome. A policy question asks whether the organisation should have acted that way.
An execution trace may answer the first question: the model produced a score above a threshold, then the payment workflow held the invoice. It does not show that the invoice was fraudulent. A feature attribution may help answer the second question, but it does not establish that a supplier’s bank-account change caused fraud or justified the payment delay.
Write down the claim before choosing an investigation method. If the issue is an incorrect input, examining the source and transformation may be more useful than generating another chart. If the issue is an unexplained pattern across suppliers, a single local explanation will be insufficient. The business guide addresses how to turn these purposes into requirements before deployment.
Reconstruct the Actual Decision Path
Begin with the case record. Identify the original invoice, extracted fields, account information available at the time, preprocessing, model version, output, policy rules, and any human actions. Distinguish what was proposed from what was executed. A correct prediction can still feed an incorrect decision if a downstream threshold or identifier is wrong.
For example, the invoice might state $1,200 while a parsing error produces $12,000. An explanation that correctly identifies the amount as influential would help locate the symptom. Establishing the cause requires tracing that value back through extraction and validation to the source. The explanation can contribute to debugging without completing the investigation on its own.
Use the relevant historical configuration where possible. Re-running today’s model against an updated supplier record may answer a new question instead of reproducing yesterday’s case. If historical reconstruction is limited, document what is missing. A current explanation should not be represented as a verified account of an earlier execution.
Include the explanation system in the record. Its version, configuration, reference data, and inputs can affect the output. Otherwise a changed explanation may be attributed to the prediction model when the change occurred entirely in the method used to describe it.
Understand What the Method Measures
LIME fits an interpretable surrogate around a prediction using sampled variations of the input. Its usefulness depends on how that neighbourhood is defined, whether the variations are meaningful, and how well the surrogate fits the model there. A nonlinear model can still be approximated usefully within a suitable region; nonlinearity alone does not make every local explanation invalid.
SHAP assigns contributions relative to a specified baseline and a definition of how feature information is included or absent. Particular algorithms calculate or approximate those contributions for particular model and value-function choices. SHAP is not solely a global ranking, and its values are not automatically causal effects. The technical explanation-methods article describes these distinctions in more detail.
Two methods can disagree because they answer different questions or use different assumptions. That disagreement is a reason to investigate their definitions and validity, rather than proof that one must be inventing a story. Local explanations for different invoices can also differ legitimately because the relevant inputs and model response differ.
Require the output to name its scope. An attribution to a model score, a local approximation, and a summary across a dataset should not all be labelled “the reason for the decision” without qualification. Clear naming prevents a valid result from being extended into a claim it does not support.
Test Faithfulness With Relevant Variations
Faithfulness concerns whether an explanation reflects the behaviour it claims to explain. For the invoice, the team could examine how the model responds to meaningful changes in amount or bank-account status while preserving other relevant conditions. The design of those tests matters: an impossible combination of fields may produce behaviour unrelated to real invoices.
Check stability where the explanation should remain stable. Correcting irrelevant punctuation should not radically change a claim that depends on financial fields, unless the preprocessing genuinely changes those fields. Conversely, a material change can reasonably alter the explanation. Stability is a property to assess against the intended meaning, not a demand for identical outputs under all changes.
Compare the explanation with independently known behaviour where available. A test model with explicit rules can reveal whether an explanation pipeline recovers the relationships it claims to describe. Such tests can expose implementation mistakes, but passing them does not prove validity for every production model. Evaluation needs cases representative of the deployed system too.
Research by Slack and colleagues on fooling LIME and SHAP demonstrated constructed models that concealed biased behaviour from particular perturbation-based explanation methods. The result supports testing assumptions and resisting reliance on one explanation channel. It does not establish that every attribution from these methods is false or that they have no diagnostic value.
Keep Model Counterfactuals Separate From Real-World Advice
A model counterfactual identifies a changed input that produces a different output under a specified model and setup. It can be evaluated against the model directly; it need not rely on a linear approximation. If changing a bank-account flag changes the invoice score, that establishes something about the evaluated model under those conditions.
It does not establish that a supplier should change its bank account, or that doing so would reduce actual fraud risk. Real-world changes can affect several variables together. Some fields cannot be changed legitimately, and some suggestions would be irrelevant or inappropriate. A useful investigation distinguishes feasible correction of a mistaken record from an intervention intended to change future circumstances.
Wachter, Mittelstadt, and Russell’s work on counterfactual explanations explores explanations intended to help people understand and respond to decisions without requiring complete access to a model’s internal structure. Turning such an analysis into practical guidance requires additional attention to feasibility, constraints, and the decision process.
The same distinction applies to feature importance. If invoice amount contributes strongly to a fraud prediction, lowering an invoice is not necessarily a valid way to remove the underlying concern. Behaviour under altered inputs is evidence about the predictor; causal advice requires evidence about the world in which someone would act.
Treat Generated Reasoning as Another Output to Evaluate
A language model can produce a step-by-step explanation that reads like an account of its internal process. The text may be helpful, but its fluency does not establish that every stated step caused the answer or that all influential information was disclosed. A later request to explain an earlier answer can introduce additional uncertainty about what the narrative represents.
Turpin and colleagues’ 2023 study of unfaithful chain-of-thought explanations tested prompting conditions in which biasing features influenced answers without being acknowledged in the explanation. This provides evidence that generated explanations can omit influences in the tested systems. It does not establish that all reasoning text is useless or that all present-day models behave identically.
For an invoice assistant, require claims to connect with checkable evidence: the relevant source passage, a verified account record, or an independently executed calculation. A generated assertion that “the amount was unusually high” should be checked against the comparison data it purports to use. If no comparison was performed, the system should not describe one as an observed step.
Tool records and verified calculations can support an explanation, but they still need interpretation. A successful database query establishes that data was retrieved, not that it was the right data for the question. Evaluate the connection between evidence, claim, and final action.
Decide What the Evidence Supports
After investigation, separate the findings. The input may have been wrong, the model may rely on an inappropriate pattern, the explanation may misdescribe its behaviour, or the policy response may be disproportionate. Several problems can coexist, and correcting one does not automatically resolve the others.
Assign the next action accordingly. A parsing defect calls for a data-path correction and review of affected invoices. A misleading explanation requires a change to the explanation process or its claims. An unsuitable hold policy needs business review even if the score and attribution are technically sound. The guardrails article explains where independent controls can limit the consequences while these issues are resolved.
Record what was established, which tests were run, and which uncertainty remains. That record makes the investigation useful for later incidents and model changes. An explanation earns its place in the decision process when its claims are defined, tested, and connected with other evidence, so reviewers can distinguish what the system did from what the organisation is prepared to justify.
