A team describes its new assistant as understanding the organisation’s expense policy. The assistant answers routine questions well, so staff begin using it to check unusual claims. One employee asks about an exception that depends on prior approval from a budget holder. The answer sounds confident but does not mention that approval requirement.
Several explanations are possible. The relevant policy passage may not have been retrieved, the instruction may have been unclear, or the model may have failed to preserve a qualification in the source. Saying that the assistant misunderstood is convenient shorthand. It becomes insufficient when the team needs to investigate the failure or decide whether the answer can be relied on.
Language about understanding, learning, and decisions can describe useful behaviour. It can also imply capabilities, memory, or authority that have not been established. The practical task is to make those implications visible and connect them to evidence about the particular system.
Separate the Question About Minds From the Application Decision
Alan Turing opened his 1950 paper with the question “Can machines think?” In Computing Machinery and Intelligence, he moved from difficulties defining the terms to a more concrete question involving an imitation game. The paper provides a useful reminder that how a question is framed affects what evidence could answer it.
A modern deployment does not settle philosophical questions about thought or consciousness. Equally, describing computation in terms of learned parameters does not by itself establish everything a system can or cannot do. A mechanism and an assessment of capability answer related but different questions.
For the expense assistant, the business needs to know whether it identifies the applicable policy, retains important conditions, and handles insufficient information appropriately. Those capabilities can be evaluated through cases with agreed expectations. The team need not settle whether the model understands in a human sense before deciding how much authority to give it.
This also avoids treating language fluency as a universal qualification. Success on routine policy questions supports a claim about those questions. It does not automatically establish competence in interpreting employment agreements, handling disputes, or authorising exceptions.
Why Human Descriptions Are So Readily Available
Anthropomorphism involves attributing human characteristics, intentions, or mental states to nonhuman entities. Epley, Waytz, and Cacioppo’s 2007 theory of anthropomorphism examines factors including available human-centred knowledge and motivations to understand other agents or seek social connection. It offers a framework for investigating these attributions, rather than a prediction that everyone will respond identically to an AI interface.
Conversational systems make human descriptions easy to use. An assistant addresses a person directly, remembers details within a session, apologises, and offers an explanation. These behaviours resemble familiar parts of conversation. A user may reasonably find the interaction helpful without knowing which parts depend on the model, stored history, retrieval, or application rules.
The design question is what users infer from that presentation. Does an apology lead them to expect a permanent correction? Does a personal tone imply confidentiality beyond the service’s actual arrangements? These are questions to investigate with representative users. A friendly interface is not inherently unsuitable, but its apparent relationship with the user should not carry promises the system cannot support.
Learning Can Mean a Parameter Change, New Context, or Stored History
When a person corrects the expense assistant, several things might happen. The current conversation may include the correction in subsequent requests. The application may save feedback for later review. A memory feature may retain a preference. The provider may eventually use selected examples in a training process. These are distinct mechanisms.
Training changes learned parameters. Supplying a revised policy or including a correction in the current context can change an answer without changing those parameters. An application can also change behaviour through new rules or retrieved information. Calling all of these learning makes it difficult to know what has been retained and who will benefit from the correction.
The organisation should explain the actual process. If feedback is reviewed monthly and does not automatically change answers, users need to know that. If a policy update takes time to reach the retrieval index, the application needs a source-currency check. A promise that the assistant learns should specify what changes and how the change is verified.
The AI terminology guide distinguishes training, inference, fine-tuning, and retrieval. These distinctions help teams decide whether a repeated error requires better source material, a workflow change, or a change to the model.
Reasoning Performance and Explanations Need Separate Evaluation
AI systems can carry out tasks involving multi-step problem-solving, search, planning, and tool use. Calling every such result mere memorisation is no more precise than treating every fluent explanation as evidence of reliable reasoning. The useful assessment examines the task, the conditions, and the correctness of the result.
A step-by-step answer can help a reviewer inspect calculations or identify an unsupported assumption. Its presence does not guarantee that the account faithfully describes the process responsible for the answer. In Turpin and colleagues’ NeurIPS 2023 study, tested models sometimes produced explanations that did not disclose the influence of biasing prompt features. The finding concerns the tested models and tasks; it shows why explanation faithfulness needs evidence of its own.
For the expense assistant, a reviewer can compare the cited policy passage with the proposed interpretation and check the relevant approval record. These checks examine evidence outside the model’s narrative. If the answer includes a calculation, the application can verify it separately.
The NIST principles for explainable AI distinguish providing explanations, making them meaningful to their audience, ensuring explanation accuracy, and recognising knowledge limits. Explanations can support debugging, education, audit, or challenge. They are not useful only when a decision can be reversed, although consequential decisions may also need a separate route to correction.
Creative Output Raises Another Question About What the Label Means
Calling an AI system creative may refer to the novelty or usefulness of its outputs. It may also suggest an intention to express an idea or an awareness of an audience. Those are different claims. A team can evaluate whether generated design options are varied, relevant, and useful without treating that evaluation as proof of a human-like creative experience.
For example, a communications team could use generated alternatives to explore how an expense policy might be explained. The reviewer would still need to check accuracy, tone, accessibility, and suitability for employees. Producing many alternatives does not establish that any particular one meets those requirements. An unusual phrase may be distracting rather than useful, while a conventional explanation may suit the task well.
The label should therefore lead to an agreed evaluation standard. In this setting, the team wants explanations that help employees apply the policy correctly. That outcome is more specific than a claim about creativity, and it gives reviewers a basis for selecting or rejecting generated material.
Agency Describes Available Actions, While Accountability Requires People
An AI agent may select tools and perform several steps towards an assigned objective. It can therefore have operational agency in a limited, defined sense. Whether it has subjective intentions is a separate question and provides no practical basis for setting access permissions.
Consider two versions of the expense assistant. One can retrieve policy and prepare an answer. Another can inspect employee records, approve claims, and initiate reimbursement. The second has materially greater authority even if both use the same language model and interface. Describing both simply as assistants conceals that difference.
OWASP’s guidance on excessive agency identifies risks associated with excessive functionality, permissions, and autonomy in language-model applications. The relevant controls apply at the tools and services that perform actions. Asking the model to respect a spending limit should not replace an authorisation check in the reimbursement system.
It is reasonable to say that an automated system made a decision when it selected an outcome. That description should still lead to identifiable responsibilities for the use case, configuration, monitoring, and correction. The organisation remains responsible for deciding where automation is acceptable and how people can challenge an outcome. Algorithmic accountability connects those responsibilities to evidence and review.
Let the Interface Describe What It Can Establish
Useful wording tells a user what has happened and what remains unresolved. An expense assistant could report that it found a policy passage requiring prior approval and that it has not checked whether approval exists. This is more informative than an assurance that it understands the claim. It helps the user determine the next action without needing to infer the system’s internal state.
Status messages should distinguish proposed and completed actions. Preparing a reimbursement request, submitting it, and confirming payment are separate events. A conversational statement of completion should be supported by the relevant system record. Otherwise natural language can imply an operational result that has not occurred.
Evaluation should include those implications as well as answer quality. Ask users what they believe the assistant remembers, what information they think it can access, and whether they believe a generated explanation has been independently checked. Compare their answers with the application’s actual design and adjust misleading presentation.
Human language will remain part of how people discuss AI. The aim is to keep it precise enough for the decision being made. Understanding should lead to a tested capability, learning to a defined update process, and agency to an explicit set of permitted actions. Those connections let an organisation use a capable conversational system while retaining clear evidence and responsibility for its effects.
