A finance team uses a model to prioritise invoices for investigation. One invoice receives a high-risk score, accompanied by a chart highlighting its amount, supplier history, and purchase-order mismatch. The chart gives the reviewer a useful starting point. It does not yet establish whether the invoice is incorrect, whether the source data is accurate, or whether the model’s learned relationships are suitable for this decision.
Explainable AI concerns methods and practices that help people understand aspects of a system’s behaviour or outputs. Those aspects vary. A developer may want to identify an unreliable feature. A reviewer may need to understand why a case entered their queue. An affected supplier may need enough information to correct a mistaken record.
An explanation should therefore be assessed against a defined question and audience. Feature attributions, simple model rules, examples, and natural-language accounts provide different kinds of evidence. Understanding those differences makes the tools more useful and their limitations easier to manage.
Begin With the Explanation the Audience Needs
For the invoice reviewer, an explanation might identify which input fields contributed to the score and link them to the source records. The reviewer can then inspect the mismatch and check whether it reflects an error or a legitimate exception. A developer may need a broader analysis of how supplier-history features behave across many cases.
Neither task necessarily requires a complete account of every intermediate computation. The required detail depends on what the person needs to establish. Conversely, a short statement that an invoice has unusual characteristics may be too vague to help either audience act.
NIST’s Four Principles of Explainable Artificial Intelligence distinguishes providing explanations, making them meaningful to their audience, ensuring explanation accuracy, and recognising knowledge limits. These are separate requirements. An accurate technical description can be incomprehensible to its intended reader, while an accessible description can misrepresent the system.
Documentation should state the explanation’s scope. Does it describe the model score, the application rule that selected the invoice, or the human decision after review? The complete decision may involve all three. A chart explaining the score does not automatically explain why a payment was delayed.
The distinction can be recorded in the review interface. It might show that the model assigned a score, a threshold routed the invoice for investigation, and a reviewer paused processing pending clarification. If the supplier corrects a purchase-order reference, the record can then show which input changed and who authorised processing to resume. This gives the team a trace of the business action alongside the model explanation.
That trace also prevents an attribution from acquiring more authority than intended. A highlighted amount may explain part of the score while the actual hold follows a separate missing-document rule. Both pieces of evidence can be accurate, yet presenting only the chart would leave the supplier with an incomplete account of the action affecting them.
Interpretable Models and Post-Hoc Methods Answer Different Questions
Some models expose a relatively inspectable structure. A small decision tree can show the sequence of conditions leading to an output. A linear model can make its coefficients available, though their interpretation still depends on transformations, interactions, and relationships among inputs. A very large tree or a heavily engineered linear model may be difficult to understand in practice.
Post-hoc methods examine an already fitted model. They may approximate its behaviour locally, allocate contributions among features, compare examples, or explore how outputs change under specified input changes. Being post-hoc does not make a method inherently misleading. It means the organisation needs to understand the relationship between the explanation and the model it describes.
Model selection should compare realistic alternatives. A simpler model may perform adequately and be easier to inspect; a more complex one may provide a material benefit. Neither a universal accuracy penalty for interpretability nor a fixed choice between accuracy, transparency, and cost can be assumed in advance.
The evaluation should consider the intended use, relevant error costs, operational effort, and the explanation requirements. If a simpler model meets those requirements, its transparency can be a practical advantage. It still needs validation of its data, assumptions, and outcomes.
LIME Approximates Behaviour Around a Particular Case
Ribeiro, Singh, and Guestrin introduced LIME as a way to explain a prediction using an interpretable model fitted locally around it. The method perturbs an input, obtains predictions from the original model, and fits a simpler explanation weighted towards the neighbourhood of the case.
The relevant question is how well that local approximation represents the original model in the region being described. Results can depend on the perturbations, neighbourhood, representation, and complexity permitted in the explanation. A locally useful account should not be presented as a complete description of global behaviour.
For the invoice model, perturbations need careful interpretation. Combining a supplier’s history with values that would never occur for that supplier may produce implausible examples. The analyst should examine whether the generated neighbourhood supports the question being asked and whether small changes in settings materially alter the explanation.
An approximation can still reveal a useful defect, such as dependence on a field that should not affect prioritisation. That finding calls for further investigation of the model and pipeline. It does not establish that every highlighted feature is an appropriate reason to act on the invoice.
SHAP Allocates Contributions Under a Defined Comparison
Lundberg and Lee’s SHAP framework uses Shapley-value ideas to assign feature contributions to an individual prediction. The interpretation depends on the value being explained, the reference or background data, and how predictions are defined when some features are treated as absent.
SHAP does not universally assume that features contribute independently. Features can interact, with contributions allocated under the chosen formulation. Dependence between features does matter: different ways of handling absent features and their relationships can answer different questions. Some estimation approaches use independence-related approximations, which should be documented rather than attributed to every SHAP method.
An additive attribution display is not a claim that the original model itself is additive. Nor does a contribution automatically identify a causal effect. A field can carry information related to another field, and the attribution depends on the defined comparison. Changing that field in the real world may have a different effect from the mathematical comparison used by the explainer.
Methods also differ in computational precision. Tree-based explanation research by Lundberg and colleagues develops efficient algorithms for Shapley-based explanations of tree models. Such methods can compute exact values for particular formulations; this contradicts the idea that explanations simply fail for all ensembles. Exact computation within a formulation still leaves the organisation responsible for choosing and interpreting that formulation appropriately.
Test Faithfulness, Stability, and Usefulness Separately
Faithfulness concerns how accurately an explanation represents the aspect of model behaviour it claims to describe. Stability concerns how the result responds to changes in inputs or explanation settings. Usefulness concerns whether the intended audience can use the explanation for its task. A favourable result on one dimension does not establish the others.
For the invoice workflow, evaluation could compare the local approximation with the original model on suitable nearby examples, repeat explanations with documented settings, and inspect cases with known data defects. Disagreement between methods should prompt investigation of their questions and assumptions, rather than an automatic vote for the most convincing chart.
User testing needs more than a satisfaction score. Can reviewers identify incorrect source data? Do they distinguish a feature contribution from proof that the invoice is wrong? Does the explanation help them reject a misleading recommendation? Test correct and incorrect recommendations so that increased acceptance is not mistaken for improved judgment.
A generated narrative introduces another layer to check. If an application converts attributions into prose, the prose must preserve their meaning and limitations. An account of what would change a prediction also needs feasibility checks; suggesting a historical record be different does not give the supplier an action they can actually take.
Explanation Requirements Depend on the Decision and Its Context
Legal obligations should be established for the particular jurisdiction and use case. A feature-importance chart is not an automatic certificate of compliance. Equally, it is inaccurate to assume that regulators require only an explanation’s existence and disregard its substance.
In its 2025 judgment in Dun & Bradstreet Austria, Case C-203/22, the Court of Justice of the European Union addressed meaningful information about the logic involved in automated decision-making in a credit-assessment dispute. The Court described an explanation of the procedure and principles actually applied that enables the person to understand how their data was used. Merely communicating an algorithm was insufficient. That ruling should not be generalised into an identical requirement for every AI output.
Beyond legal requirements, the invoice team needs ownership of explanation quality. Retain the relevant model version, input references, explanation settings, and business rule applied. Reassess explanations when those components change. A correct chart for an earlier model should not accompany a newer score without validation.
The guide to algorithmic accountability connects evidence with review and correction. The production safety guide develops the operational controls needed when the system behaves unexpectedly. Explanation contributes to both, provided its scope remains clear. It helps people investigate and assess a decision; it does not replace validation of that decision or responsibility for the action that follows.
