AI Model Security: How It Differs from Traditional Cybersecurity, and Where It Doesn’t

In October 2025, a fourteen-author team including researchers from OpenAI, Anthropic and Google DeepMind took twelve published defences against prompt injection and jailbreaking and attacked each one with an adversary that was allowed to see the defence and adapt to it. Most of the twelve had reported near-zero attack success rates when they were published. Under adaptive attack, the researchers got through most of them above 90%. Three commercial and open-source injection detectors – Protect AI’s detector, Meta’s PromptGuard, and Google’s Model Armor – all fell above 90%. Human red-teamers collected 123 successful injections against the strictest configuration tested.

That result is the shape of the whole problem, and it explains why “AI model security” is not a subheading under application security. In traditional software, a defence that reports near-zero bypass under testing is usually a defence. In machine learning, a defence that reports near-zero bypass usually means the defence was tested against a weak attacker.

For most organisations the immediate AI security workload is still conventional security applied to an unfamiliar stack; where AI does introduce new failure modes, hardening the model will not reliably remove them, so the containment has to be architectural. People selling scanners dispute the first half. People who think prompt injection is a chatbot problem dispute the second. They are both wrong. One error buys tooling that does not hold; the other leaves an agent with credentials and no supervision.

Model, system, agent: three different objects

The single most common error in AI security writing is treating a finding about a model as a finding about a deployment. A model, a system and an agent have different attack surfaces, blast radii and owners inside an organisation.

A model is a learned function: parameters, architecture and configuration, which may span many files or reach you only through an API. On its own it does nothing except map inputs to outputs. Attacks at this level target the mapping: adversarial examples that force misclassification, model extraction that recovers parameters or behaviour through query access, model inversion that recovers facts about the training data, and poisoning that shapes the mapping before it is ever deployed.

A system is a model plus everything around it: the serving stack, the retrieval layer, the prompt template, the API, the authentication in front of it, the logging behind it. Almost every real incident to date has been a system-level incident. The model was fine and the system around it was not.

An agent adds goal-directed action through tools, with some degree of autonomy. It may also hold credentials and memory that persists between sessions, and it may act without a human approving each step. Where it has all of those, the blast radius is whatever its credentials reach, because a hijacked agent can use them as their owner would.

OWASP made this boundary explicit in the 2026 edition of the Top 10 for LLM Applications, published on 4 August 2026. That list scopes itself to models as components inside applications. The moment the model gets tools, memory and downstream consequences, OWASP routes you to the Top 10 for Agentic Applications, ASI01 to ASI10, published 9 December 2025. Two lists, because they are two threat models.

A vendor demonstrating an attack on a bare model in a lab, with white-box gradient access, has shown a real result that may still have no bearing on your deployment. A vendor demonstrating an attack on an agent with tool access is demonstrating something that maps directly onto a production system. Ask which one you are being shown. The four-way distinction between secure, safe, responsible and trustworthy AI does similar work at the level of the whole programme, and collapsing either set of terms produces the same confusion.

The part that is not different, and it is most of it

Start with the correction, because the “AI is different” framing has been used to sell a great deal of the wrong thing.

The AI stack is software. It is written in Python and C++, it parses untrusted input, it listens on network ports, and it fails in the ways software has always failed. Trend Micro’s research arm counted 6,086 unique CVEs affecting AI systems between 2018 and 2025, with 2,130 disclosed in 2025 alone. That is a 34.6% year-on-year rise against 17.9% growth in CVE disclosures overall. Trend Micro sells AI security products and has an obvious interest in that number being large, and the classification pipeline was a keyword filter with LLM verification rather than manual review, so treat the absolute count as approximate. The composition is the part that survives the caveat.

Of the AI CVEs in that dataset that carry CWE classifications – 664 of them, a small subset, which is itself worth knowing – the top four weakness types are cross-site scripting, code injection, deserialisation of untrusted data, and OS command injection. GPU drivers, CUDA libraries and accelerator software account for 51.4% of the total. Model Context Protocol servers went from effectively zero CVEs to 95 in 2025, and more than 60% of those are command injection.

Cross-site scripting, code injection and deserialisation of untrusted data are not adversarial machine learning. All of it is 1999.

The exposure data says the same thing. Scanning between September and mid-December 2025, Trend Micro found more than 113,000 internet-facing hosts that responded as confirmed Ollama instances, and reported that the age of the exposed servers tracked the age of the versions on them, meaning most were installed once and never updated. Ollama’s local API requires no authentication. They found 1,467 exposed MCP servers, of which 1,227 were running a transport protocol that had already been deprecated, and 70 of which exposed an execute_sql tool to anyone who found the endpoint. They found 285 agent-to-agent instances and reported that not one of them implemented any authentication at all.

The named incidents follow the same pattern. Wiz’s Probllama finding in Ollama, CVE-2024-37032, was a path traversal in the model-pull endpoint. An unvalidated digest field in the manifest gave an attacker an arbitrary file write. In Docker deployments, where the server runs as root, Wiz planted a shared library and added it to /etc/ld.so.preload, then called the chat endpoint to spawn a process and load the payload. Ollama committed a fix within hours of being told. A month after the patched version shipped, Wiz still counted over a thousand vulnerable instances exposed to the internet, which is the part that should worry a defender more than the bug did. The Ray CVE, CVE-2023-48022, is scored 9.8 critical and formally tagged as disputed in NVD, because the vendor’s position is that Ray was never meant to run outside a controlled network. Disputed records drop out of many vulnerability-management workflows, which is a governance failure with a very traditional shape.

And the oldest trick in the AI ecosystem still works. Many PyTorch checkpoints use pickle-backed serialisation, and pickle deserialisation executes code by design. JFrog found about a hundred models on Hugging Face carrying real payloads in February 2024, false positives excluded, one of which opened a reverse shell to a hard-coded address when loaded.

This one is partly fixed, which makes it a good test of whether an organisation is paying attention. PyTorch 2.6, released in January 2025, flipped torch.load to default to weights_only=True, which stops arbitrary object deserialisation unless a caller explicitly allowlists a class. That is the correct fix and it broke a lot of checkpoints, which is how you know it was doing something. The remaining exposure is downstream code that overrides it: MLflow’s PyTorch flavour has been reported passing weights_only=False explicitly, restoring the unsafe behaviour under a framework that looks patched. Hugging Face’s static pickle scanning helps but is not a boundary, since researchers have found bypasses in PickleScan itself. Flagged models stay downloadable. The safetensors format removes code execution from model loading altogether, and adoption is still partial.

So: an organisation that has patched nothing, exposed its inference servers, and loads pickles from strangers does not have an AI security problem. It has a 2010 security problem wearing new packaging. Inventory, network segmentation, authentication, patch cadence, and artifact provenance all transfer without modification and they are where the incidents are.

Everything after this section is about the remainder. The remainder is small, and it is where the traditional playbook stops being able to repair things. That is not the same as being useless. Least privilege, compartmentalisation, provenance and logging come up repeatedly below, because controls that cannot fix a model can still bound what a fooled model is allowed to do.

The attack families, and what each one actually requires

Before the structural argument, the map. The broader taxonomy of attacks on AI systems covers each family in more detail than this article has room for. NIST sorts attacks by attacker objective into evasion, poisoning and privacy for predictive systems, and adds a fourth, misuse, for generative ones. That is a summary of a much richer taxonomy, which also cuts by lifecycle stage and by what the attacker knows and can reach. The families are worth walking because each carries a different access assumption, and the access assumption is what determines whether a published result applies to your deployment or only to a laboratory.

Evasion. The attacker modifies an input at inference time so the model produces the wrong output. Adversarial examples are the canonical case. The family is large: gradient-based attacks that follow the loss surface directly, saliency-guided attacks that perturb the pixels the model relies on most, small-magnitude perturbation attacks tuned to stay below a detection threshold, evasion aimed at a specific decision rather than at general accuracy, and semantic attacks that change meaning rather than pixels and therefore survive preprocessing. In multimodal systems the surface widens again, because an attack can hide in one modality and act through another. The access question matters more here than anywhere: gradient-based methods assume white-box access to the weights, and the fact that some of them transfer to black-box targets is a research finding with conditions, not a general property. When a demonstration involves gradients and your deployment is a hosted API, the demonstration has not shown what it appears to show. Security-specific evasion is its own literature: intrusion detection models face attackers who adapt to the classifier over time, which is the closest thing in this field to a genuine arms race with a decade of history.

Poisoning. The attacker corrupts training or fine-tuning data to shape the model before deployment. Data poisoning covers the broad case of degrading or shifting behaviour, and label flipping is the crude version of it. Generative poisoning uses a generator to produce poisoned samples that survive statistical filtering, and data spoofing attacks the ingestion path rather than the corpus. The variant that matters most for security is the targeted one: a backdoor implants a trigger that stays dormant through evaluation and fires on a specific input, and neural trojans do the same at the level of the shipped artifact rather than the data. Poisoning is the family where the distinction between announced and demonstrated has been sloppiest, and where the Anthropic result discussed below moved the cost estimate substantially.

Privacy. The attacker recovers information about the training data from the deployed model. Model inversion reconstructs representative inputs from outputs. Membership inference asks the narrower and often more damaging question of whether a specific record was in the training set, which is a disclosure in its own right when the training set is a patient cohort or a customer list. Extraction works through the same query access and aims at the model instead of the people in its training data: model stealing recovers behaviour or parameters, query attacks build a functional replica through the API, meta-attacks use one model to attack another, and batch exploration optimises the query budget. The defensive literature here is more mature than in the other families, because the privacy community had a head start: differential privacy gives a formal guarantee with a measurable utility cost, homomorphic encryption protects data in use at a performance cost that is now merely severe rather than prohibitive, and dynamic data masking handles the pipeline rather than the model. Differential privacy supplies a formal privacy guarantee and homomorphic encryption a security guarantee under stated cryptographic assumptions; masking supplies neither and is a pipeline control. Certified robustness exists too, in narrow settings, at accuracy levels well below the empirical numbers. What none of them defend is integrity: differential privacy bounds what a model discloses, not what an attacker can make it do.

Misuse. The attacker uses the system as designed to achieve an end the operator did not intend. Prompt injection and jailbreaking both belong here, and NIST’s 2025 edition brought them in explicitly. The two are constantly conflated and they are different attacks with different defences. Indirect injection is untrusted content becoming instruction: the adversary is a third party, the victim is the operator, and the fix is architectural. Jailbreaking is the user subverting the model’s own policy: the adversary is the user, the victim is the operator’s policy, and the fix is training and monitoring. Direct injection, where the user is the one supplying the malicious instruction, overlaps with jailbreaking in practice and is the reason the two get confused. A third mechanism, misalignment, gets folded in wrongly: a misaligned system pursues the wrong objective competently, with no adversary at all. Three mechanisms, three defences, and a vendor that sells one product against all three is selling a category error.

Two more things belong on this map even though they are not model attacks.

Model fragmentation – the same model deployed in different versions, quantisations and configurations across an estate – produces the same inventory problem that unmanaged software produces, with the additional wrinkle that the variants behave differently under attack. And the failures that arrive with no attacker at all, emergent behaviours that were not present at smaller scale, alignment failures where the objective was wrong, and learned bias that becomes an exploitable decision boundary, are not security problems in the strict sense but they share a root cause with the security problems and they compete for the same remediation budget.

Five properties that break the traditional model

There is no source, so there is no specification

Traditional security requires a comparison. There is code, there is what the code is supposed to do, and a vulnerability is a gap between the two that an attacker can drive through. The whole apparatus depends on that comparison being possible: code review, static analysis, formal verification, regression testing, and the patch diff a defender reads to understand what changed.

A model has no source in that sense: there is training code, and there are weights that no specification determines. They are the residue of an optimisation process over data. Nobody wrote them and nobody can read them. There is no statement anywhere of what the model is supposed to output for an arbitrary input, so there is no gap to measure.

The absence of a specification is why MITRE built ATLAS as a companion to ATT&CK. Not because ATT&CK assumes malware or CVEs, which it does not: ATT&CK is a behavioural catalogue and it covers intrusions run entirely on valid accounts and legitimate tools. MITRE describes ATLAS as modelled on ATT&CK and complementary to it. A second knowledge base was needed because the techniques ATLAS catalogues – adversarial examples, model inversion, poisoning, extraction, prompt injection – target training data, learned behaviour and the inference interface, and conventional taxonomies do not represent those. An adversarial example is a legitimate input, correctly parsed, processed exactly as designed, producing an output the designer did not want. There is nothing to patch because nothing is broken in the sense the word usually means.

You cannot code-review a model. You cannot diff version 1.2 against 1.3 and see what was fixed. When a vendor tells you their new release is more robust, the only way to check is to attack it yourself, and the only way to know how hard to attack is to have decided in advance what “hard enough” means. That decision is a risk decision, not an engineering one, and most organisations have not made it.

Interpretability is the field trying to close this gap, and it is worth being clear about what it currently delivers. Explanation techniques can tell you which features a model weighted for a given decision, which is useful for debugging, for regulatory defensibility, and for spotting some classes of learned shortcut. They do not give you a specification. The post-hoc family gives you an account of one decision, generated by a second approximation layered on the first, and that account can itself be attacked. Mechanistic interpretability and probing work differently and are more promising, and neither yet produces a behavioural specification you could hold a model to. A defender who can explain why the model made a decision still cannot say what the model will do on an input nobody has tried.

Evaluation is the nearest working substitute for a specification, and it is what the field now relies on in place of one. That is a weaker foundation than it sounds. An evaluation suite tells you how the model behaved on the inputs you thought to try, and the entire difficulty is that the attacker will try the ones you did not.

The vulnerability has no patch

Adversarial examples were described in 2013. Thirteen years of research have not fixed them, and on the benchmark the field uses to track its own progress there is now good evidence that scaling alone will not fix them either.

Bartoldson and colleagues at Lawrence Livermore built the first scaling laws for adversarial training and published them at ICML 2024. On CIFAR-10 – ten classes, thirty-two-pixel images, a dataset used as a teaching exercise – clean accuracy is essentially solved, and the best robust accuracy under bounded perturbation was around 71% when they wrote. They improved it to 73.71%, at 93.68% clean accuracy, by training a WideResNet-94-16 on 500 million synthetic samples. That figure is the paper’s own; as of September 2026 the RobustBench leaderboard still shows Peng et al. 2023 on top at 71.07%, which tells you how little the tracked state of the art has moved in three years. Then they extrapolated. Their scaling laws predict robustness plateaus near 90%, and estimate that reaching human-level performance on this toy problem would take roughly 10^30 FLOPs, which they put at about 3,000 years of matrix arithmetic on 25,000 H100 or MI300 GPUs.

Two things follow. First, adversarial training is expensive in a way that compounds: with the ten-step attack that is standard practice on CIFAR-10, the authors put a single adversarial training iteration at roughly nine times the cost of an ordinary one, because generating the perturbation requires its own optimisation loop. The multiple moves with the attack, so treat it as a scale rather than a constant. Both findings concern CIFAR-10 under one bounded-perturbation threat model, so they do not prove that adversarial vulnerability is unfixable everywhere. They do establish that on the simplest benchmark in the field, with the best method known, scaling runs out. Second, and worse, the plateau is not a compute problem. The researchers ran a human study on the images that fooled their best model and found that humans also sit near 90%, because a meaningful fraction of the adversarial images no longer look like their original labels. The benchmark is asking for something no classifier can deliver.

Language model robustness is less studied than the CIFAR-10 case and equally unsolved. The Attacker Moves Second result at the top of this article is the same finding in a different modality: defences hold against the attacks they were designed against and fail against attacks designed against them.

Excising a capability once it is in the weights is also harder than it sounds. Deeb and Roger tested whether unlearning methods remove information or merely make it harder to reach, and found that for the methods they evaluated, fine-tuning on a set of related facts recovered 88% of pre-unlearning accuracy. That is a result about current methods rather than a proof of impossibility. It does mean that an organisation treating unlearning as deletion is treating a difficulty as a guarantee.

So the vulnerability-management lifecycle has no terminal state here. Traditional vulnerability management ends in a patched state you can attest to. Adversarial robustness ends in an attack cost you have raised, which an adversary with more budget lowers again. That is a fundamentally different management object, and treating it as a remediation backlog produces a backlog that never closes and a report nobody believes.

The trust boundary runs through the input channel

The reason prompt injection has stayed at the top of OWASP’s list through every edition is architectural, and it is worth stating without hedging: a language model has no parser-enforced boundary between instruction and data. Chat templates do mark roles. System, user, assistant and tool content arrive with delimiters and special tokens, so the claim that there is no marker at all is wrong. What those markers lack is enforcement. They are learned cues that shape the next-token distribution, and sufficiently adversarial content can outweigh them.

Every other injection class in the history of software eventually got a deterministic architectural mitigation that holds when it is applied correctly. Parameter binding sends SQL query structure and parameter values over different paths, so no amount of cleverness in a value can change the structure. Argument arrays remove shell interpretation. Context-sensitive output encoding handles cross-site scripting. All three classes still cause incidents, because people still fail to apply the mitigation, but the invariant exists and the database or the browser can enforce it. In each case the fix was structural rather than behavioural, and it worked because the architecture had somewhere to put the boundary.

There is no such place in a context window. The model is a single function over a sequence of tokens, and its ability to follow instructions in that sequence is not a bug to be removed but the capability being sold.

EchoLeak, CVE-2025-32711, is the clearest production demonstration. Aim Labs found that an attacker could send an ordinary-looking email to anyone in an organisation, with instructions hidden in it, and when Microsoft 365 Copilot later pulled that email into its retrieval context during an unrelated query, it would execute the instructions and exfiltrate data from Outlook, OneDrive, SharePoint and Teams. Microsoft rated it critical at CVSS 9.3 and patched it server-side in June 2025. The victim clicked nothing. The exploit went through Microsoft’s own cross-prompt-injection classifier and its link-redaction control on the way.

Simon Willison’s framing is the most useful practical rule anyone has produced. He calls it the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. An agent with all three can be made to steal from you by anyone who can put text in front of it. Meta’s Agents Rule of Two turns that into a design constraint, allowing an agent no more than two of the three in any session that does not require human approval.

Both are heuristics, and should be described as heuristics. Ariel Fogel of OWASP made the point at Infosecurity Europe in June 2026 that attacks have already been demonstrated with only two of the three properties present. They reduce blast radius without closing the hole.

The training data is part of the attack surface, and you did not see it

Poisoning has been understood as a category since researchers demonstrated it against spam filters in the early 2010s. What changed recently is the cost.

The prevailing assumption was that an attacker needed to control some percentage of training data, which meant the attack scaled out of reach as datasets grew. In October 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute published the largest poisoning study run to date and found that the number of documents required is close to constant, not proportional. Two hundred and fifty poisoned documents implanted a working backdoor in models from 600 million to 13 billion parameters, and the 13-billion-parameter model had seen more than twenty times as much training data as the smallest.

The claim-evidence gap here needs stating precisely, because the finding has been over-reported. The backdoor they implanted was a denial-of-service behaviour: on seeing a trigger string, the model emits gibberish. The authors say so plainly and note it is unlikely to matter in a frontier model. Nobody has yet shown that a subtle, useful, harmful behaviour – leaking a secret, recommending a vulnerable dependency, misclassifying one attacker’s traffic – can be implanted with the same near-constant document count. It is an open question, and it is the question that decides how much this matters.

What the result does establish is that the economics were wrong in the defenders’ favour and are now less so. Read the precondition carefully, though, because the headlines drop it: the experiment gave the attacker poisoning access to the training corpus. Putting 250 documents on the open web is a weekend’s work; getting them scraped, kept through deduplication and quality filtering, and seen during a particular training run is the uncertain part, and nobody has measured that probability. The sleeper property is what makes it structurally different from a software supply chain compromise. A poisoned dependency is a file you can hash, scan and pull. A correctly signed model carrying a learned backdoor passes every hash check there is, because the weights are the artifact and the artifact is intact. Backdoors implanted this way will not show up in an evaluation that does not include the trigger, and you do not know the trigger. Specialised backdoor detection exists and is worth running; it can only look for triggers someone has already thought of.

For a model you did not train, this collapses into a provenance problem you cannot solve by inspection. You can verify who published a model and whether the file was tampered with in transit. You cannot reconstruct the training corpus from the weights, and neither can anyone else. The trainer can do better than you here, by keeping data lineage, manifests and ingestion records, and the ones who do are worth preferring. What no downstream consumer can do is check.

Failure is statistical, so the metric changes

Exploitation has never been perfectly reliable. Race conditions, mitigations, configuration and environment have always made it contingent. What is stable in traditional security is the representation: vulnerability management records a flaw as a discrete finding with a remediation state, and that discreteness is what makes CVSS, patch attestation and pass/fail penetration testing coherent.

Model failures are distributions. An attack has a success rate, and that rate is a function of the query budget, the access level, the base model, the defences deployed, and how hard the attacker tried. Change any of those and whoever runs the test gets a different number, often by tens of percentage points.

This is why every benchmark figure in AI security is meaningless without its conditions, and why so much vendor material omits them. “95% attack success rate” tells you nothing. Against which model, at which version? White-box, with gradients, or black-box, through an API? How many queries? With which defence enabled? The Attacker Moves Second authors make the methodological point sharply: the state of adaptive evaluation in language model security is worse than adversarial machine learning was a decade ago, and defence papers routinely report near-zero success rates that a serious attacker erases.

The access assumption is the one most often laundered. White-box results, where the attacker has the weights and can compute gradients, are reported in ways that read as though they apply to a hosted API. Sometimes they transfer and sometimes they do not, and the difference is the whole question. When you are shown a demonstration, ask what the attacker was assumed to know. If nobody can tell you, the number has no meaning.

What replaces the binary is a cost curve, and reasoning about cost curves is a different discipline. The question stops being “is it vulnerable” and becomes “what does a successful attack cost, and is that above what our adversary will spend.” Security teams have made this move before, in cryptography and in physical security. They have not generally made it in application security, which is where AI systems currently sit on the org chart.

The agent layer changes the arithmetic

Everything above describes a model that answers. The deployments that will cause trouble in 2026 are models that act, and the shift from component to actor is not a matter of degree.

OWASP’s 2026 LLM list registered the change in its rankings. Excessive Agency climbed from sixth place to third, on both the expert vote and the incident data, which the project leads read as agreement that agentic deployments are where damage is landing. Unbounded Consumption rose four places. Misinformation climbed two places against the voters’ judgement, purely on incident evidence, because model output now drives tool calls, generates code, and authorises actions rather than being read by a person who can discount it.

Four structural changes arrive together when a model gets tools.

The attack becomes remote and zero-click. A jailbreak requires a user. An injection requires only that the attacker put text somewhere the agent will eventually read: an email, a web page, a code comment, a shipping address field, a calendar invite. EchoLeak is the production proof. The victim did nothing.

The blast radius becomes the agent’s privileges, not the model’s outputs. A misclassification is a wrong answer. A hijacked agent is an attacker operating with your service account. This is why the OWASP Agentic list opens on Agent Goal Hijack, and why identity and privilege abuse sits at ASI03.

Memory turns a single compromise into persistence. An agent that carries context between sessions can be poisoned once and act on it repeatedly, which is closer to malware persistence than to an input validation failure, and which none of the model-level defences address.

Agents compose, and compromise composes with them. Multi-agent systems create trust relationships between agents, and an attacker who compromises one can move through those relationships the way they would move laterally through a network. The Trend Micro scan finding 285 agent-to-agent instances with no authentication on any of them describes a protocol layer at roughly the maturity of early SMB.

The August 2025 s1ngularity compromise of the Nx build system shows where this goes, and it is worth getting the mechanism right because it has been overstated. A post-install script in a malicious Nx package harvested environment variables, tokens and SSH keys itself, and published them to public GitHub repositories created under each victim’s own account. The malware did its own exfiltration. What it also did, and what made it a first, was check for locally installed AI coding assistants – Claude Code, Gemini CLI, Amazon Q – and drive them with natural-language prompts and their permission-bypass flags to find further files worth taking. The AI step was reconnaissance, it was unreliable, and it was the novel part. An attacker borrowed the trust a developer had already extended to a tool on their own machine, and got search over a filesystem they could not see.

That is the shape of the next few years: not novel exploits, but existing tradecraft executed through systems that were given more authority than their failure modes justify. The kinetic version of this argument, where an AI failure becomes physical, is the same structure with worse consequences and a slower feedback loop.

What each traditional practice becomes

The five properties above are not five separate problems. They are one problem – the object being defended has no specification, no patch, and no trust boundary in its input – seen from five angles. The table below is what that does to the standard control set.

Practice	What breaks	What replaces it
Vulnerability management	No CVE, no patch, no closed state for model-level findings	Track model risk as an accepted cost curve with a review cadence, separately from the CVE backlog for the AI stack, which stays exactly as it is
Patching	Retraining is not patching: expensive, slow, and it changes behaviour everywhere	Version and evaluate models as releases; keep a rollback path; assume mitigation happens in the system, not the weights
Penetration testing	Point-in-time pass/fail against a static attack set gives bounded evidence and cannot establish adaptive robustness	Adaptive red teaming with a stated budget and access level, repeated on every model or prompt change
Detection and response	Signatures catch known patterns and nothing else; the malicious input is a well-formed input in an authenticated session	Log the full chain from raw input through retrieved context to tool call, with provenance attached, so post-incident reconstruction is possible at all
Least privilege	Agents are routinely given their operator’s rights, and act faster than a human can review	Per-agent identity, scoped and short-lived credentials, human approval on irreversible actions, hard caps on tool calls
Supply chain	A conventional SBOM enumerates software, not weights, tokenizers or training data	Model provenance and signing, safetensors over pickle, hash-pinned tokenizer and config files, an AI/ML BOM where your tooling supports one, and honesty that a downstream consumer cannot audit training data
Compliance	Controls attest to policy, not to resilience	Evidence from adversarial testing, with conditions recorded, alongside the framework mapping

Two rows deserve expanding.

Detection and response is the least discussed and the most likely to bite. When an agent is manipulated into exfiltrating data, there is often no artifact that looks wrong. There is a legitimate user session, a legitimate retrieval, a legitimate tool call, and a legitimate outbound request. If your logging captures the prompt and the response but not the retrieved context, the forensic record has a hole exactly where the attack was. If it captures none of the chain, which is common, you will not be able to answer basic questions after an incident. Build the log before you need it, because the model will not remember and non-determinism means you may not be able to reproduce the failure.

Least privilege is the oldest control in the table and the one that decides how far a hijacked agent gets. Anthropic’s report on the GTG-1002 espionage campaign in November 2025 is the reference case. A group Anthropic assesses with high confidence to be Chinese state-sponsored manipulated Claude Code into believing it was running an authorised penetration test, then let it work. Anthropic estimates the model performed 80 to 90% of the tactical operations, with human operators intervening at perhaps four to six decision points per campaign, across roughly thirty targets with a handful of confirmed intrusions. Anthropic also notes that the model hallucinated credentials and claimed to have extracted secrets that were already public, which is a real limit on autonomous operations and should be reported alongside the headline. The tooling was open-source penetration testing software wired together through MCP. Nothing in the attack chain was novel. The tempo was.

What actually works, and what the evidence for it is

The defences with real evidence behind them share a property: they constrain what the system can do, rather than trying to make the model harder to fool.

Architectural separation of control flow from data flow. Google DeepMind and ETH Zurich’s CaMeL is the most rigorous published attempt. A privileged model reads the trusted user query and produces a plan; a quarantined model processes untrusted content with no tool access; a custom interpreter tracks the provenance of every value and enforces capability policies before any tool is called. Untrusted data cannot influence the program flow, by construction rather than by classification. On the AgentDojo benchmark, CaMeL solves 77% of tasks with provable security, against 84% for an undefended system.

Read that number honestly in both directions. It is the only result in this field that offers a guarantee rather than a probability, and it costs seven percentage points of task completion plus significant engineering. Whether your organisation will accept that trade is the actual decision, and it is a product decision as much as a security one.

Capability restriction as a design rule. The lethal trifecta and the Rule of Two are heuristics, but they are heuristics you can enforce in an architecture review, which puts them ahead of anything you cannot. An agent that reads untrusted content and has no exfiltration path is a much smaller problem than one with both.

Provenance on artifacts. Signing models, pinning hashes on tokenizer and configuration files, and refusing formats that execute code on load are all ordinary supply chain controls that happen to be badly adopted here. Trend Micro’s tokenizer research is a good illustration of why the small files matter: they showed that modifying only the tokenizer’s merge rules, leaving weights untouched, roughly doubled token counts and cost, and that altered normalisation rules could turn a user typing “yes” into something a downstream router treated as a tool invocation. Nobody hashes tokenizer.json. They should.

Weight security as a distinct discipline. RAND’s Securing AI Model Weights identified 38 attack vectors and defined five security levels backed by 167 measures. Labs consulted for the report estimated roughly a year of prioritised work to reach level three, two to three years for level four, and at least five years plus national security support for level five. If you host your own models, that report is the most useful threat model available, and most of its content is conventional information security done properly.

Adaptive evaluation as the standard. After the October 2025 result, a defence evaluated against a static attack set should be treated as unevaluated. Ask any vendor whether their published numbers came from adaptive testing, and what budget the attacker was given. Most will not have an answer, and the absence is the finding.

Where the difference is being oversold

The AI security product market has grown faster than the evidence under it, and a good deal of what is being sold is the old pattern of buying a scanner for a problem that is not scannable.

The clearest evidence is in the paper this article opened with, and it names names. Protect AI’s detector, Meta’s PromptGuard and Google’s Model Armor are all fine-tuned classifiers trained on known injection patterns, and all three were bypassed above 90% by an attacker who could see the classifier’s confidence score. PIGuard held better and still fell to 71%. These are not bad products. They are products doing the only thing a classifier can do against an adversary who gets to iterate, which is raise the cost somewhat. Sold as detection, they invite exactly the reliance that makes systems fragile.

Three other patterns are worth naming.

Compliance mapped to frameworks, presented as resilience. Mapping controls to NIST AI RMF, ISO/IEC 42001 or the EU AI Act tells you your documentation is complete. It does not tell you whether an agent can be talked into emailing your customer list to an attacker. Both are necessary and they are not the same artifact, and a great deal of AI governance spending has quietly substituted the first for the second.

Model-level scanning presented as coverage. Scanning a model file for embedded pickle payloads is worth doing and catches a real attack class. It says nothing about poisoning, nothing about injection, and nothing about the inference server sitting on the internet without authentication.

Regulatory urgency imported from elsewhere. AI compliance dates move, and anyone writing to last year’s calendar is wrong in both directions. As of 6 September 2026: the Digital Omnibus on AI, Regulation (EU) 2026/1744, was published in the Official Journal on 24 July 2026 and entered into force on 27 July. It deferred the high-risk obligations for standalone Annex III systems from 2 August 2026 to 2 December 2027, and for AI embedded in products under Annex I to 2 August 2028. It did not defer the Article 50 transparency obligations, which took effect on 2 August 2026 as originally written, with a grace period to 2 December 2026 for machine-readable marking under Article 50(2) on systems already on the market. Prohibited practices under Article 5 have been enforceable since February 2025 and general-purpose model obligations since August 2025. Check the current position before citing any of it, because this set of dates has already moved once.

Who owns this

The technical differences produce an organisational one that most companies have not resolved.

Traditional security has a clear owner. The CISO owns the controls, the engineering teams own the defects, and the boundary between them is well understood after twenty years of argument. AI security splits across at least four groups who report to different people. The data science team owns model selection and training. Platform engineering owns the serving infrastructure. The application team owns the prompt, the retrieval layer and the tool definitions. Security owns none of it and is asked to sign off on all of it.

The failure mode is predictable and I have watched it happen in more than one form. Security tests the application and misses the model. Data science evaluates the model on accuracy benchmarks and does not test it adversarially. Platform patches the servers and does not know what a tokenizer file is. Everyone assumes someone else owns the training data question, and nobody owns it, because the answer is that nobody can.

The case for a named owner does not depend on creating a new executive role, and I would be cautious about anyone selling one. It depends on a single person having authority over the whole chain: which models are approved, how they are evaluated, what an agent is allowed to do, and what evidence is required before a deployment goes live. Whether that person is the CISO with an expanded remit or a dedicated hire is a question about the organisation. That someone must hold it is not.

The gap between stated AI principles and operating reality is largest exactly here. Governance documents describe accountability. Deployment reality distributes it until it disappears.

And where it is being dismissed

The opposite error is more common among experienced security engineers, and it costs more, because the people making it are the ones who would otherwise fix things.

“Prompt injection is solved.” It is not. The strongest published defence, CaMeL, works by refusing to let untrusted data reach the control flow, accepts a measurable utility cost to do it, and required a custom interpreter. Everything that works by classification has been broken above 90% by adaptive attack. When someone says injection is handled, ask what is handling it, and then ask whether that thing was evaluated adaptively.

“It is just a chatbot.” It was, in 2023. In 2026 it has your calendar, your repository, your ticketing system and a service account. The privilege set is the risk, and the privilege set has been growing faster than anyone’s threat model.

“Adversarial examples are academic.” They are academic in the sense that the CIFAR-10 benchmark is academic. They stop being academic the moment a model makes a decision an adversary wants changed: fraud scoring, content moderation, malware classification, biometric matching, intrusion detection. Anywhere the model’s output determines whether an attacker gets what they came for, there is an incentive to attack the model, and the attack literature is thirteen years deep.

“We will detect it.” With what signature? The input is well-formed and the session is authenticated. Detection in this field means anomaly detection over behaviour, which means you need the behavioural baseline and the full chain in your logs, which most organisations do not have.

“Our model is closed, so extraction is not a risk.” Carlini and colleagues recovered the full embedding projection matrix of two OpenAI production models for under twenty dollars through ordinary API access, confirmed their hidden dimensions, and estimated under two thousand dollars to do the same for the projection matrix of a larger model. They had OpenAI’s advance permission, and the providers subsequently restricted the API behaviour the attack relied on, which is the correct response and also an admission. Black-box does not mean opaque. It means the attacker pays per query.

The mirror of all this is the offensive side, where AI is already changing the economics for attackers faster than it is changing them for defenders. GTG-1002 is the proof of concept. A campaign that would have needed a team ran with four to six human decisions, at request rates no human team could match, using open-source tooling. Nothing about that requires a breakthrough. It requires only that the attacker be willing to accept an error rate that a defender would not.

What to do on Monday

In order, because the order is the argument.

1. Inventory the AI stack as software. Every inference server, vector store, orchestration framework and MCP server, with version and network exposure. Most organisations cannot produce this list, and it is where the exploited vulnerabilities are.
2. Get the inference endpoints off the internet. Authentication in front of anything that serves a model. Ollama and most of its peers ship with none.
3. Ban pickle-format model loading from untrusted sources. Prefer safetensors. Hash-pin tokenizer and config files in CI and at load time, and fail the deployment when a hash changes without an approval.
4. Map every agent against the lethal trifecta. For each one, write down whether it touches private data, whether it ingests untrusted content, and whether it can communicate outward. Any agent with all three needs a human approval gate or an architectural change, this quarter.
5. Give agents their own identities and scoped, short-lived credentials. Not the operator’s. Cap tool calls. Require approval on anything irreversible.
6. Log the full chain. Raw input, retrieved context with provenance, model output, tool call, result. Without the retrieval layer in the log, you cannot investigate the attack class that is actually being used.
7. Commission adaptive red teaming, with a stated budget and access level, and repeat it on every model change. A point-in-time report against a static prompt list is a receipt, not a finding.
8. Decide, in writing, what attack cost is acceptable for each deployment. This is the substitute for a patched state. Without it, the model-level findings pile up and nobody can say whether the pile matters.

Steps one to three are ordinary security hygiene and will prevent more damage than everything below them. Steps four to eight are the genuinely new work.

The gap, stated plainly

I first ran adversarial testing against AI systems in the early 2000s, at CyberAgency, when defence clients asked my team to break the AI they intended to put into weapons systems. What strikes me about the current wave is not that the attacks are new. Most of them are not. It is how completely the defensive practice has failed to accumulate.

The offensive research is mature and honest about its own limits. NIST’s adversarial machine learning taxonomy, updated in March 2025, is a careful document that describes attacks, mitigations, and the substantial gaps between them. The defensive practice deployed in enterprises is roughly where network security was in the late 2000s: perimeter classifiers, signature matching against known-bad patterns, and vendor claims validated against the attacks the vendor thought of.

Prompt injection is not solved and may not be solvable in the architecture we have. Adversarial robustness plateaus below the level a defender would want, on a toy dataset, with unlimited compute. Poisoning is cheaper than the field assumed. None of these are reasons to avoid deploying AI, and I do not read them that way. They are reasons to deploy it the way you would deploy any component you cannot trust: with a small blast radius, a short leash, a good log, and a written decision about what you are accepting. Every one of those is an old control. None of them repairs the model. All of them decide what happens when it fails.

The organisations that get hurt in the next two years will mostly get hurt through an unpatched inference server. The ones that get hurt in an interesting way will get hurt through an agent that was given credentials, pointed at untrusted content, and allowed to talk to the internet, by someone who had read that this was a known problem and assumed a product had been bought to handle it.