A company is considering a model that recommends which customer accounts should receive an additional review. The supplier offers an explanation dashboard showing the factors behind each score. During the demonstration, the display looks clear. The unanswered questions concern the work around it: can staff check those factors, can customers correct inaccurate information, and can the company reconstruct an earlier recommendation after the supplier updates the model?
For business leaders, explainable AI is an investment and operating requirement. The organisation needs to specify what people must be able to understand, why that understanding matters, and what evidence will demonstrate it. Buying an explanation feature without those decisions can leave the business with an attractive dashboard and an unresolved operational problem.
Specify the Use Before the Explanation
Begin with the role of the model in the business process. Does it suggest which accounts to investigate, determine an eligibility outcome, or simply organise information for a person? A recommendation can still influence consequential decisions, especially when reviewers accept it routinely. The assessment should follow actual use rather than the supplier’s product category.
Identify affected people and the actions that may follow. In the account-review example, a flag might lead only to an internal check, or it might delay access to a service. Those consequences change the evidence, review, and communication requirements. The accountability article explains how decision rights and escalation should follow these responsibilities.
Then define the questions an explanation needs to answer. A reviewer may need to know which account records contributed to a flag. A customer may need to identify an incorrect transaction. An investigator may need the historical policy and model version. These requirements should inform system selection before the organisation commits to a design that cannot support them.
Separate the Audiences and Their Decisions
Different recipients do not necessarily need separate technical platforms, but they do need information suited to their purpose. A feature-attribution chart might help an analyst investigate an unexpected pattern. It is unlikely to be a complete customer notice or an adequate description of the overall decision process.
NIST’s Four Principles of Explainable Artificial Intelligence distinguishes providing an explanation, making it meaningful to its audience, accurately reflecting the system’s process, and recognising knowledge limits. These distinctions help leaders avoid accepting readability as the only test. A clear statement can still misdescribe the system, while an accurate technical output can be unusable for the intended recipient.
| Recipient | Business requirement to establish |
|---|---|
| Frontline reviewer | Check the relevant evidence and change or escalate a recommendation |
| Affected customer | Understand the outcome, inspect relevant facts, and request correction or review |
| Technical owner | Investigate model behaviour and failures in data or integration |
| Approval authority | Assess whether performance, limitations, and controls support the proposed use |
Use this as a starting point for discussions with the people doing the work. The requirement should describe an activity they can complete, not merely an artefact the supplier can deliver. “Identify the source of an incorrect account balance” is more testable than “provide a transparent interface.”
Compare Alternatives on the Whole Requirement
Ask the team to compare a conventional rule-based process, an interpretable model where appropriate, and more complex candidates. Evaluate performance on relevant data, including the kinds of error that matter operationally. Complexity does not establish quality, and interpretability does not automatically require lower accuracy. The result depends on the task and available models.
Interpretability also depends on presentation and scale. A small decision tree may be easy to inspect; a very large one may not be. A linear model has an explicit mathematical relationship, but transformations, interacting features, and unfamiliar units can still make its output difficult for users to understand. The complete pipeline deserves assessment.
For a more complex model, ask which explanation methods are being proposed and why they fit the requirement. Local attributions, counterfactual examples, and summaries of overall behaviour answer different questions. The technical guide to explainability methods explains their assumptions and limitations in more detail.
The comparison should include implementation and maintenance. A modest performance gain may be worthwhile if it changes valuable outcomes and the organisation can operate the system responsibly. It may be insufficient when explanation costs, review effort, integration, or unresolved limitations outweigh the benefit. The decision needs evidence about this use, rather than a preference for either simplicity or sophistication in principle.
Ask Suppliers to Demonstrate Difficult Cases
A sales demonstration usually shows selected examples. Request an evaluation using representative cases chosen with your team, including incomplete records, disputed facts, unusual circumstances, and cases the system should decline to assess. Establish permitted data use before sharing sensitive material. A controlled evaluation can use appropriately prepared records without granting broad production access.
For each case, ask the supplier to connect the input, prediction, explanation, and action taken by the application. Determine whether the explanation comes from the deployed model, a separate approximation, a rule, or generated text. Ask what configuration or reference data influences it and how it is tested. A single “explainable” label conceals too much variation.
Include known problems that the explanation should help uncover. If an account balance is deliberately wrong in a test record, can the reviewer locate the source and identify a correction? If the system lacks required information, does the explanation acknowledge that limitation? These tests examine the business requirement directly instead of rewarding the persuasiveness of a narrative.
Ask for evidence of performance across relevant case types and groups, while handling sensitive data appropriately. A clean-looking explanation does not establish fair treatment or predictive validity. Those questions need their own evaluation, informed by the context and applicable obligations.
Make Historical Evidence a Procurement Requirement
Disputes rarely arrive conveniently before a release changes the system. The organisation needs a way to establish what happened at the relevant time. Determine which versions of the model, preprocessing, policy, explanation method, and source records can be identified for an individual case.
Historical reconstruction may depend on the supplier retaining an old model or providing a sufficiently complete record of the original output and execution. Do not assume a current API can reproduce an earlier decision. Stochastic generation and changing dependencies can complicate replay. Define what evidence will be preserved, for how long, and under whose access controls.
Put update expectations into the agreement where necessary. Material changes should have notification, evaluation, and approval arrangements proportionate to their effect. Clarify access to documentation, incident support, export formats, termination assistance, and the information available for responding to a complaint. Supplier confidentiality can be addressed through appropriate arrangements without leaving operational responsibilities undefined.
Avoid promising unlimited logging as the solution. Records can contain sensitive personal or commercial information. Retention, access, deletion, and security requirements belong alongside auditability. The procurement decision should establish how those requirements will work together.
Treat Legal Requirements as Specific Requirements
There is no universal rule that attaching a SHAP chart makes an AI system compliant. Obligations depend on the jurisdiction, sector, data, role of automation, and consequences for affected people. Legal assessment should identify which notices, explanations, review rights, and records the particular process requires.
The Court of Justice of the European Union’s 2025 explanation of its judgment in C-203/22 concerns access to meaningful information about automated credit assessment under the GDPR. It emphasises describing the procedure and principles actually applied in a way that enables understanding and challenge. Providing an algorithm alone was not treated as sufficient. That is a scoped legal example, not a general certification test for every AI system.
Translate applicable requirements into acceptance criteria and operating procedures with the responsible legal and process owners. A customer service team needs to know how a request will be recognised, what information can be supplied, and who can conduct review. The explanation feature must support that process rather than stand in for it.
Budget for Use, Evaluation, and Maintenance
Implementation costs include more than explanation computation. Teams may need to prepare source information, design reviewer interfaces, test comprehension, maintain documentation, train staff, and handle requests for review. The AI ROI article describes how to distinguish these costs from assumed benefits.
Assign ownership for monitoring whether explanations remain accurate and useful. Model changes, revised policies, new data sources, and changing users can affect what the explanation means. A customer notice retained from an earlier release may no longer describe the current process, even if its wording is still easy to read.
Measure outcomes connected to the requirement. Can reviewers find material input errors? Can customers identify the relevant records and obtain correction? Can an investigator reconstruct a disputed case? Satisfaction and adoption can be useful additional measures, but higher confidence alone does not establish that the system deserves greater reliance.
Set an Approval Boundary the Business Can Defend
The final decision should state the permitted use, evidence reviewed, limitations, required controls, and conditions for reassessment. If the explanation cannot support a necessary review activity, the organisation can narrow the use, add independent evidence, choose another model, or delay deployment. Approval need not pretend every uncertainty has been eliminated.
Business leaders should be able to explain why the complete process is suitable for its purpose and who will act when it fails. That requires performance evidence, workable controls, and clear responsibility alongside explanation methods. A well-specified explainability requirement contributes to that decision by making the necessary understanding concrete enough to demonstrate and maintain.
