A manager asks an AI assistant whether a proposed expense is permitted. The answer is fluent, specific, and supported by what appears to be a policy reference. A useful response is to ask what “permitted” means here: under which policy, for which employee, on what date, and with whose approval? The answer may change as these conditions become explicit.
This way of examining an answer has an affinity with Socratic questioning. It does not require imagining that Socrates predicted AI or would have endorsed a particular product. The connection is a method of inquiry: clarify the claim, examine the reasons offered, look for contradictions, and identify what remains unresolved. Applied carefully, it can improve the work people do with AI without turning philosophical language into a technical evaluation method.
Begin With the Historical Distinction
Socrates left no writings of his own, and accounts of his thought come through other authors. Plato’s dialogues present a Socrates who questions claims to knowledge, but they should not be treated as a verbatim record of every historical conversation. When applying them to AI, distinguish the ancient text from a modern interpretation.
The familiar line “I know that I know nothing” is a compressed paraphrase, not the wording of Plato’s Apology 21d. In that passage, Socrates contrasts someone who thinks he knows what he does not know with his own refusal to assume such knowledge. The Jowett translation of the Apology gives the discussion in its wider context.
The useful connection concerns recognising the limits of a claim. It does not establish that Socrates regarded every expert as ignorant or that practical skill has no value. Nor does it settle whether a modern machine can know or understand something. Those are further questions that need arguments of their own.
For the expense example, the practical lesson is modest: a confident answer should be examined against the reasons and evidence relevant to the decision. This applies to the manager’s assumptions as well as to the assistant’s output.
Define the Words Carrying the Argument
Many disagreements survive because a central term changes meaning during the discussion. An expense can be “eligible” for reimbursement while still requiring prior approval. A system can be “accurate” on a selected test while remaining unsuitable for a particular workflow. An AI-generated answer may use these words naturally without establishing which interpretation applies.
Ask for a definition tied to the task. In the expense case, permitted might mean that the current policy allows this category, the employee falls within its scope, the amount is within a limit, and the correct approver has authorised it. The organisation must decide whether these conditions are sufficient, rather than letting the assistant invent missing rules.
Plato’s Euthyphro provides a historical example of questioning a proposed definition and examining its implications. It is a philosophical dialogue about piety, not a design specification for software. The modern application is to examine whether the definition being used actually answers the question and remains consistent as examples change.
Carry the agreed meaning into the work. If “approved” means a recorded decision by an authorised manager, a generated sentence saying an expense looks reasonable should not be presented as approval. Clear definitions connect language with a verifiable organisational state.
Separate Reasons From Reassuring Language
An assistant may say the expense is acceptable because it is business-related. That is a proposed reason, but its relevance depends on the policy. Business purpose might be necessary without being sufficient. The manager should ask which rule connects that fact with the conclusion and whether additional conditions apply.
Request the source, then inspect it. A policy reference should lead to the relevant passage, with an identifiable version and scope. A citation can be genuine while failing to support the claim, and a model can produce an incorrect reference. Asking for evidence is useful only if someone or something checks what the evidence actually says.
Distinguish an observed fact from an inference. “The receipt records a meal for four people” is different from “the meal was necessary for the project.” The latter may require context not present in the documents. Keeping that distinction visible prevents an unverified assumption from becoming a fact through repetition.
Generated explanations also need evaluation. Turpin and colleagues’ study of unfaithful chain-of-thought explanations showed that biasing features could influence answers without being acknowledged in the explanations under the tested conditions. The finding supports caution about treating an explanation as a complete record of why an answer was produced; it does not imply that every generated argument is invalid.
Use Counterexamples to Test the Rule
Once the assistant states a rule, vary a condition that should matter. What if the same expense occurred before the employee joined? What if the amount exceeded the limit? What if the person claiming reimbursement was also the proposed approver? These cases can expose a rule that sounded reasonable but omitted an important requirement.
The counterexample should be relevant to the actual decision. A long sequence of increasingly unusual hypotheticals can consume time without improving the policy interpretation. Choose cases from real ambiguity, known exceptions, or plausible failures. Compare the answer with the authoritative rule or a qualified decision maker’s interpretation.
Ask what would change the conclusion. A useful response might identify a missing approval record or a policy exception that needs confirmation. This makes uncertainty actionable. A vague statement that “circumstances may vary” leaves the original question unresolved without showing what evidence is needed.
Record the cases used to settle the interpretation. They can later inform tests, staff guidance, or a policy clarification. The value comes from the resulting evidence and decisions, not from making the assistant concede that its earlier wording was wrong.
Consistency Is Necessary Evidence, Not a Guarantee
An answer that contradicts an earlier statement deserves investigation, but a consistent answer can still be wrong. Two claims may share the same mistaken assumption, and a longer explanation can repeat that assumption more persuasively. Socratic questioning should therefore include independent checks, not only requests to reconcile the conversation.
Keep track of what changed. If new information legitimately alters the answer, revision is appropriate. If the assistant agrees with whichever position the user most recently proposed, the apparent resolution may reflect the interaction rather than better evidence. Test the material claim against its source instead of rewarding agreement.
Passing an exam or solving a benchmark task is evidence of performance under particular conditions. It is not proof of competence in every related situation, but neither should it be dismissed as meaningless pattern matching. The question is how well the evaluated task represents the intended use and what further evidence is required.
The explanation evidence guide develops this distinction for technical investigations. A clear argument, a correct answer, a faithful account of model behaviour, and a justified business action are related achievements with different evidential requirements.
Turn Uncertainty Into an Operating Decision
Epistemic humility in an AI-assisted process should be visible in what happens when the evidence is insufficient. The assistant might ask for the applicable policy, identify a missing record, or direct the case to an authorised reviewer. Saying “I may be wrong” while still presenting an unsupported decision provides little practical protection.
Systems can include abstention, confidence estimation, and rules that require further checks. Their effectiveness needs testing against the intended task. It is inaccurate to claim that AI can never decline to answer or that uncertainty estimation is only beginning to be studied. It is equally inaccurate to assume that a polite disclaimer demonstrates reliable recognition of knowledge limits.
For the expense process, define the unresolved conditions that block an answer or action. If the policy version is unknown, retrieve the current one. If an exception requires judgment, route it to the responsible person. The guardrails article explains how such boundaries can be enforced beyond the wording of a prompt.
There also needs to be a stopping point for questioning. Once the required evidence has been checked and the authorised person has resolved the relevant issue, continuing indefinitely may add little value. When the issue cannot be resolved, stop the proposed action and record the uncertainty rather than prolonging the dialogue until it produces a convenient answer.
Apply the Questions to the Organisation Too
A manager can interrogate an assistant carefully while leaving the organisation’s own assumptions untouched. Why is this policy being applied? Does the organisation have the necessary records? Is the proposed automation suitable for the consequences? Who can correct a mistaken outcome? These questions examine the process in which the answer will be used.
The human oversight article considers whether the people reviewing recommendations have enough evidence, time, and authority. An instruction to question the model cannot compensate for a reviewer who lacks access to the policy or is required to accept recommendations at an unworkable rate.
Socratic questioning is useful when it produces clearer definitions, stronger evidence, and an explicit account of what remains unknown. It should sharpen the organisation’s judgment about AI-assisted work, while leaving technical testing, professional expertise, and accountable decisions with the processes that can actually provide them.
