An AI readiness assessment examines whether an organisation can develop, acquire, use, and operate an AI application under defined conditions. It is useful when it identifies requirements and evidence for a particular decision. It becomes less useful when a general score is treated as permission to deploy any system described as AI.
Readiness depends on the use case and stage. A team may be ready to explore a document assistant with approved sample material while lacking the permissions, evaluation, or support needed for live use. Another organisation may have strong shared infrastructure but no suitable data for a proposed prediction task. Neither is adequately described by a single ready or not-ready label.
The assessment should therefore state what decision it supports: whether to investigate an idea, run a bounded pilot, release a service, or expand its authority. The result is an evidence-based account of what can proceed, under which limits, and what remains unresolved.
Define the Use Case and Compare the Existing Process
Consider a procurement team evaluating an assistant that extracts information from supplier documents and prepares a comparison for review. Its intended benefit is to reduce preparation effort while preserving the evidence buyers need to assess options. Selecting a supplier and approving a contract remain outside the proposed authority of the assistant.
That scope determines the assessment. The team needs to know which document types are included, which fields matter, who reviews them, and what happens when information is missing or conflicting. It also needs to identify the people affected by an incorrect comparison and the decisions that might follow from it.
Establish the current process and plausible alternatives. Better document templates, a conventional extraction tool, or improved source management may address part of the problem. The assessment should not assume that a model is the preferred answer before examining the actual work.
Record a baseline that includes preparation, checking, correction, and follow-up. If the assistant produces a comparison quickly but buyers spend longer tracing unsupported claims, the relevant benefit is the combined effect. The pilot guide explains how to evaluate that workflow under bounded conditions.
Separate Essential Conditions From General Maturity
An organisation-wide assessment can identify shared needs such as identity management, procurement capability, or staff training. Those findings help plan investment. They do not replace the requirements of a specific application.
For the procurement assistant, some conditions may be essential before real documents can be processed. The organisation needs permission to use the material in the chosen environment, controls that restrict access appropriately, and a defined review path. Strong infrastructure elsewhere cannot compensate for an unresolved condition that blocks this use.
Other gaps may be compatible with a limited experiment. A small evaluation set might support exploration while being insufficient evidence for broad release. Limited integration may be acceptable for an offline test but impractical for daily work. Describe the permitted stage and limitations explicitly.
An illustrative assessment could organise the evidence as follows:
| Area | Evidence to inspect | Decision it informs |
|---|---|---|
| Purpose and ownership | Defined task, baseline, affected people, and decision owner | Whether the proposal is worth investigating |
| Data and access | Representative sources, permitted use, and enforced permissions | What information can be processed and by whom |
| Evaluation | Agreed criteria and independently reviewed examples | Whether the output is useful enough for the intended role |
| Workflow and people | Review design, staff capacity, training, and escalation | Whether the process can be operated as proposed |
| Operations and change | Monitoring, support, release evidence, and recovery | Whether use can continue safely as conditions change |
Scores can summarise a discussion, but retain the evidence, uncertainty, and unresolved conditions beneath them. An average obscures more than it clarifies if readers cannot see that a critical permission or recovery requirement is missing.
Inspect the Data Path and the Evaluation Method
Readiness is not established by the existence of a data warehouse or document repository. The procurement team needs sources that are relevant, accessible, sufficiently current, and interpretable for the task. Supplier names, document versions, units, and effective dates may need reconciliation before a meaningful comparison is possible.
Determine how the application obtains that information. A pretrained model with retrieval has different requirements from a custom-trained classifier. The assessment should identify which components need to be developed, which are purchased, and which claims depend on supplier evidence that may be unavailable.
Build evaluation cases with people who understand the procurement work. Include ordinary documents, missing fields, conflicting statements, and material qualifications that should not be omitted. Agree what a correct extraction means and when the system should leave a field unresolved. A polished comparison is insufficient if its claims cannot be traced to the supplied material.
Keep development examples separate from the cases used for the final assessment of a proposed version. Repeatedly tuning against the same test cases weakens their value as independent evidence. Examine relevant error types and source categories rather than relying only on an aggregate result.
The assessment should also record what the evaluation cannot establish. A small set of familiar suppliers may provide little evidence about a new language or document format. Uncertainty can justify a narrower scope or more investigation; it should not be hidden in a confidence score that has not been validated for this task.
Test Integration, Authority, and Review as Working Arrangements
The assistant’s output needs a usable place in the procurement process. Can buyers open the source passage? Can they correct a field without losing its provenance? Does a revised document replace an earlier version visibly? If the tool requires repeated manual copying between systems, measure that effort before accepting the expected savings.
Access requirements should be enforced by the application and connected services. A user should not gain access to another team’s restricted supplier material simply because the interface accepts a natural-language question. Inputs from documents should not grant the assistant authority to take actions outside the approved workflow.
OWASP’s guidance on excessive agency identifies risks from giving language-model applications excessive functionality, permissions, or autonomy. For this assistant, preparing a comparison does not require permission to approve purchases or alter supplier records. Excluding those capabilities can make the initial scope easier to evaluate and contain.
Reviewers need time, relevant information, and authority to resolve or escalate discrepancies. Training should address the actual cases they will encounter, including unsupported statements and missing qualifications. A demonstration of the interface is not evidence that reviewers can detect consequential errors during normal workload.
Name the people who can approve use, change its scope, and pause it when necessary. The accountability guide develops those decision rights. The assessment should test whether the named owners can carry out their responsibilities, including obtaining help and resources across team boundaries.
Verify the Capacity to Operate and Reassess the Service
Operational readiness includes expected volume, response time, recurring cost, dependency availability, and support. Follow the full path through retrieval, model processing, validation, and review. A fast model endpoint cannot compensate for a source system that does not provide the required information reliably.
Define the evidence needed when a comparison is disputed. The team may need the source version, output actually reviewed, relevant configuration, reviewer changes, and subsequent action. Retention and access arrangements should support investigation while limiting unnecessary duplication of sensitive documents.
Specify how a material change is evaluated. A provider update, a new document parser, or an expanded source collection can affect the output even when the visible interface remains the same. Readiness should be revisited when the conditions supporting approval change, rather than assumed to persist indefinitely.
The NIST AI Risk Management Framework places risk management across the lifecycle and includes post-deployment monitoring and response. Those expectations help organise an assessment, but the team must supply evidence about its own service, owners, and operating conditions.
Fallback also needs a practical test. If the assistant is unavailable, can buyers continue the most important work using the existing process? If a faulty extraction affects comparisons already prepared, who finds them and arranges reassessment? Restoring the software does not automatically repair work that depended on earlier outputs.
Turn the Findings Into a Bounded Next Step
The report should separate verified conditions, assumptions, open questions, and blocking gaps. Give each unresolved item an owner, an evidence requirement, and a decision date or trigger. A generic recommendation to improve governance is less useful than identifying which approval or review route must be demonstrated before a pilot begins.
The next step may be an experiment designed to resolve one uncertainty. It may be a supervised release, an alternative approach, or a decision that the expected benefit does not justify further work. Treat each as a legitimate finding when it follows from the evidence.
An AI readiness assessment cannot guarantee delivery or eliminate future uncertainty. It can make the current decision more credible by identifying the requirements, evidence, authority, and capacity needed for the proposed use. Its value becomes visible when the findings change what the organisation authorises and what it does next.
