A security analyst receives an alert about an unusual sign-in. An AI assistant can gather related events, summarise the account’s recent activity, and suggest a containment action. That can save time. If the summary omits a legitimate administrative change or the assistant can disable accounts without adequate checks, the same acceleration can interrupt important work.
Attackers can use similar capabilities to research targets, adapt messages, and inspect software. The resulting contest is often described as AI versus AI, but the systems making decisions include people, credentials, data sources, business processes, and enforcement tools. A model’s capability is only one part of the outcome.
The practical question is where AI reduces useful work, where it introduces uncertainty, and which actions the surrounding system permits. Answering that question requires separating observed use, demonstrated capability, and forecasts about future operations.
Treat Threat Assessments as Assessments
The UK’s National Cyber Security Centre assessment of AI’s impact to 2027 judges that AI will increase the effectiveness and efficiency of elements of cyber intrusion. It discusses areas including reconnaissance, social engineering, vulnerability research, and code generation. These are assessed developments, rather than a measured success rate for every attacker or organisation.
That distinction matters when interpreting demonstrations. A model finding a flaw in a selected test program shows a capability in that setting. It does not establish that the same system can reliably compromise an unfamiliar production network with its access controls, monitoring, dependencies, and operational constraints.
Equally, the absence of a dramatic new technique does not make the change unimportant. Reducing the time required to personalise messages or examine candidate weaknesses can affect attack volume and defender workload. The security implication can arise from making established steps cheaper rather than inventing an entirely new attack category.
Avoid presenting an inevitable winner. Defenders can exploit knowledge of their own systems, centralised policy enforcement, and opportunities to block a whole class of operations. Attackers can seek unevenly protected paths. Which advantage matters depends on the target, objective, access, and controls in place.
Verify Requests Independently of Their Presentation
Generative systems can help produce fluent messages and plausible impersonations. A request to change a supplier’s payment details should therefore be evaluated through the approved change process, even when its language and apparent sender look familiar. Fluency was never reliable proof of authority.
The durable control is an independent route for confirming the consequential request. Staff can use a previously established contact method and the organisation’s approval process, rather than a new telephone number included in the same suspicious message. The purpose is to verify the change through evidence the sender of that message does not control.
Authentication also has layers. A legitimate account can be compromised, and a genuine employee can be mistaken about the request. Verifying who sent a message does not necessarily establish that the proposed payment or privilege change is appropriate. Business authorisation still needs to be checked.
AI-based message classification can assist triage, but a confidence score should not substitute for these controls. The classifier may help prioritise review while the workflow independently enforces who can change banking details or grant access.
Use Defensive AI Where the Evidence Can Be Checked
Security operations contain substantial summarisation and correlation work. An assistant can assemble a timeline from logs, explain why an alert fired, group related observations, or draft an investigation query. These tasks can be useful when the output retains references to the underlying events.
A summary should distinguish observed facts from inferences. An unfamiliar address, a changed device, and a successful login are observations. Concluding that an account was taken over is an interpretation requiring more context. Analysts need a route to inspect the original records and identify missing telemetry.
Code analysis offers another example. A model can identify a suspicious path and propose a fix, but the finding should establish reachability, relevant permissions, and the conditions needed for an impact. A plausible explanation of vulnerable-looking code is not the same as a validated vulnerability in the deployed system.
Measure the work saved after verification. If an assistant generates many convincing but incorrect findings, analysts may spend more time investigating them than they save reading logs. The value lies in useful evidence and accepted findings, not the volume of generated analysis.
| AI-assisted task | Useful output | Check before consequential action |
|---|---|---|
| Alert summarisation | Referenced timeline and missing evidence | Original events and scope |
| Vulnerability analysis | Specific path and conditions | Reachability and reproducibility |
| Incident triage | Prioritised hypotheses | Alternative explanations |
| Containment recommendation | Proposed target and expected effect | Authority, business impact, and recovery |
Keep Evidence Separate From Instructions
An assistant reading emails, tickets, web pages, or repository files encounters material controlled by other parties. That material can include instructions intended to redirect the assistant. If the application treats retrieved content as authority to call tools, the information source becomes a route to action.
NIST’s adversarial machine-learning taxonomy describes attack categories and terminology, including threats to generative systems. The application-level concern is concrete: an attacker-controlled document should not decide which credentials the assistant uses or which data it sends elsewhere.
Enforce access in the surrounding tools and services. A model asked to summarise one user’s tickets should receive only the information that user is authorised to access. Tool operations should validate arguments and permissions independently of a model’s explanation that the action is legitimate.
A prompt telling the model to ignore malicious content can contribute to defence, but it is not a complete boundary. Limit tool capabilities, separate sensitive credentials from untrusted processing where possible, and test realistic attempts to cross those boundaries. A document can be useful evidence without becoming an instruction source.
Match Response Authority to Consequence
Automating an observation is different from automating an intervention. Adding context to an alert generally has a different consequence from blocking a customer network, revoking credentials, or deleting a suspected malicious file. The action policy should recognise that difference.
For well-defined cases, a tested automated playbook may be appropriate. It needs clear eligibility conditions, bounded targets, an audit record, and a way to handle uncertainty. A broad instruction to take whatever action appears necessary delegates policy decisions that the organisation may never have examined.
Human review is useful only when the reviewer can inspect the relevant evidence and understands the proposed effect. Showing a persuasive paragraph and an approve button can preserve a nominal approval step while making independent assessment difficult. Provide the source events, affected resources, and expected recovery path.
Restoring service may also be more involved than undoing a command. Revoked credentials can break active integrations, and quarantining a system can interrupt a business process. Test containment and recovery together so the cost of a false positive is understood before a high-volume incident.
Evaluate Failures That Ordinary Demonstrations Miss
Use an evaluation set containing confirmed incidents, benign administrative activity, incomplete telemetry, and ambiguous cases. Include attacker-controlled content if the assistant reads external material. Measure missed incidents and false positives separately; a single accuracy percentage can hide the cost of each.
For an illustrative batch of 10,000 cases containing 100 real incidents, detecting 90 percent of incidents yields 90 true alerts. Incorrectly flagging one percent of the 9,900 benign cases adds 99 false alerts. Fewer than half of the resulting 189 alerts represent incidents. These assumed rates are not a product benchmark; they show why evaluation needs realistic incident prevalence and review capacity alongside detection performance.
Check behaviour when tools fail or return stale data. An unavailable identity service should not silently become evidence that an account is malicious. A failed containment request should not be reported as successful simply because the assistant attempted the call.
Test response permissions with accounts of different privilege levels. The same natural-language request should not gain additional access because it is phrased confidently or appears inside a retrieved ticket. Keep a durable record of actions, tool results, and the authority under which they occurred.
AI services also need ordinary operational controls. NCSC’s secure AI development guidance covers security across design, development, deployment, and operation. Models, dependencies, data flows, and integrations all belong in that lifecycle; securing the prompt alone leaves much of the application unexamined.
Judge the Result at the Incident Level
Track time to a confirmed finding, time to effective containment, analyst review effort, and the disruption caused by incorrect actions. Faster alert closure is not necessarily an improvement if unresolved incidents are being dismissed or legitimate users are being blocked.
Record model and workflow changes with evaluation results so regressions can be investigated. A new model or expanded tool permission can alter both capability and risk. Human oversight needs to follow those changes rather than being treated as a one-time approval of the original demonstration.
AI can make parts of attack and defence substantially faster. A dependable defensive workflow uses that speed to improve evidence gathering and response while retaining enforceable limits around access and action. That gives the organisation something concrete to evaluate, rather than a promise that purchasing another model will settle the contest.
