Robust Optimization Is the Only Defence That Held, and Only Inside Its Own Threat Model

On 10 October 2025, researchers from OpenAI, Anthropic, Google DeepMind and ETH Zürich published attack results against twelve defences for language models. Most of the twelve had reported near-zero attack success in their original papers. Under attacks built specifically against each defence, the numbers inverted. Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Florian Tramèr and ten co-authors reached 100% attack success against Circuit Breakers and StruQ, 96% on MetaSecAlign, and above 90% on most of the others. Their paper, The Attacker Moves Second, has since been accepted at USENIX Security 2026. Human red-teamers in their competition defeated every challenge they were given.

One paragraph of that paper names the exception. Training against a fixed set of precomputed or weak perturbations does not generalise and falls quickly to adaptive attack. Only adversarial training that performs robust optimization, where the perturbation is optimized inside the training loop, has ever produced measurable robustness, and only over the space of attacks the optimization actually searched.

That is the whole subject of this article, and the qualifier at the end of it does most of the work. Robust optimization is the one model-level defence with a record of surviving attacks designed to break it, across the twelve years since Szegedy and Biggio described adversarial examples in 2013. Robust optimization survives on terms that most buyers would refuse if a vendor stated them plainly: one perturbation set, declared in advance by the defender; a clean-accuracy cost; a compute cost measured in multiples rather than percentages; and, on CIFAR-10, a 32×32 pixel ten-class problem, a best measured robust accuracy of 73.71%, with the best provable number closer to 40%.

The formulation, and what the formulation assumes

Aleksander Madry and colleagues stated the problem as a saddle point in Towards Deep Learning Models Resistant to Adversarial Attacks, published at ICLR 2018. Minimize, over the model parameters, the expected value of the maximum loss over all perturbations inside a set S. Two nested optimizations: an inner one that searches for the worst input the attacker could produce, and an outer one that updates the weights against whatever the inner search found.

The framing predates deep learning by decades. Abraham Wald’s minimax decision rules from the 1940s and the robust optimization programme built by Aharon Ben-Tal, Laurent El Ghaoui and Arkadi Nemirovski all solve the same shape of problem: optimize for the worst case inside a declared uncertainty set, rather than the average case over an assumed distribution. Madry’s contribution was to show that the shape held for neural networks, and that projected gradient descent was a good enough inner solver to make it trainable.

Every robustness claim in this field is a statement about one declared set S and about nothing outside it. It is not discovered from data. Somebody writes it down. On CIFAR-10 the conventional choice is an L∞ ball of radius 8/255, meaning every pixel may move by at most about 3.1% of its dynamic range; RobustBench uses 4/255 for ImageNet. Those radii became standard because they are tractable and because they produce numbers comparable across papers, not because attackers restrict themselves to them.

This is also the line between robust optimization and data augmentation, and the line matters more than its size suggests. Augmentation samples perturbations from S. Robust optimization searches S for the worst one and trains on that. Empirically, that difference is the entire result. I have covered the attack side of this in adversarial attacks on AI models and in gradient-based attacks; the defensive side is the same optimization, solved by the defender during training.

Training on attacks is not robust optimization

The most common error in defensive machine learning is to mix a set of known attacks into the training data and then call the result adversarially trained. The model learns that set and nothing about the space it was drawn from.

Anish Athalye, Nicholas Carlini and David Wagner demonstrated the cost of that error in 2018. Examining the nine non-certified white-box defences accepted at ICLR that year, they found seven relying on obfuscated gradients and circumvented six completely and one partially within each paper’s own threat model. The one defence that held was Madry’s, which was also the only one in the group that solved the inner maximization during training rather than making it hard for an attacker to solve at test time.

Seven years later, the same error produced the same result in a different domain. Circuit Breakers, StruQ and MetaSecAlign all generate adversarial examples using heuristics and optimization procedures, then fold those examples into training. All three fell to Nasr and colleagues. MetaSecAlign reported 2% attack success on the static AgentDojo benchmark and reached 96% under a search-based adaptive attacker. StruQ failed in every case they tested. The authors are explicit that this repeats what the vision literature already established.

So the first question to put to any robustness claim is not whether the model was adversarially trained. It is what the inner maximization was. If it is a fixed dataset of attack strings, the method is data augmentation and the robustness claim covers only those strings.

What it costs on the scoreboard

RobustBench standardises robust-accuracy evaluation on AutoAttack, an ensemble of white-box and black-box attacks that Francesco Croce and Matthias Hein built to stop each new paper from choosing its own generous evaluation, and records a lower figure wherever an adaptive attack finds one. Reading the leaderboard on 6 September 2026:

• A standard WideResNet-28-10 on CIFAR-10 reaches 94.78% clean accuracy and 0.00% under AutoAttack at ε = 8/255. Not low. Zero.
• The best entry is Brian Bartoldson and colleagues’ WideResNet-94-16 from Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies, ICML 2024, at 93.68% clean and 73.71% robust.
• On CIFAR-100 at the same radius the best is 42.66%. On ImageNet at ε = 4/255 the best is 59.68%, from MIMIR on a Swin-L backbone, against 0.00% for a standard ResNet-50.

Two things about that leaderboard deserve attention. The first is that the top CIFAR-10 entry is an ICML 2024 paper and nothing on the leaderboard has displaced it since. Almost two years of a static ceiling on a dataset with sixty thousand images and ten classes.

The second is what reaching 73.71% required. The models at the top of that board are trained with enormous volumes of synthetic data: Sven Gowal’s group used 100 million DDPM-generated samples, and Zekai Wang and colleagues swapped the DDPM for an EDM diffusion model to push it further. Bartoldson’s team derived scaling laws for adversarial training and found that existing state-of-the-art methods burn excess compute for the robustness they achieve; their compute-efficient setup beat the prior record with 20% fewer training FLOPs and a 70% reduction in inference FLOPs. Their scaling laws also suggest the asymptote for this problem is around 90%, not 99%.

The cheap version of adversarial training is much cheaper and much worse. Eric Wong, Leslie Rice and Zico Kolter showed in Fast is better than free that FGSM with random initialization trains a CIFAR-10 model to 45% robust accuracy in six minutes on a single GPU. RobustBench records that method at 83.34% clean and 43.21% robust. The same paper names the failure mode that had defeated single-step adversarial training before it: catastrophic overfitting, in which robust accuracy collapses to zero within a single epoch if the step size is slightly wrong, while clean accuracy keeps climbing and the training curve looks healthy.

The perturbation set is the claim

Every one of those numbers describes one shape of perturbation. Change the shape and the guarantee disappears.

RobustBench’s own tutorial makes the point with the CIFAR-10-C corruption set. On fog at severity 5, a standard WideResNet reaches 74.4% accuracy. Three well-known L∞-robust models on the same images reach 38.8%, 31.1% and 22.0%. The models hardened against adversarial perturbation are worse than the undefended one against weather. Adversarial training does help on average across the corruption set, but the exceptions are not small and they are not rare.

Perturbation size behaves the same way. Evaluating the top RobustBench defences, one group found almost no accuracy left at ε = 20/255, a perturbation still small enough that a person would not call the image damaged. No L∞ ball contains the semantic transformations I have written about under semantic adversarial attacks, or the multi-channel attacks covered in multimodal attacks.

Bartoldson’s group added an uncomfortable finding to this. They showed adversarial images that fooled their state-of-the-art model to human subjects, and the humans often agreed with the model’s new label. If a perturbation inside the declared budget can change what the correct answer is, then part of the residual 26% is not a defensive failure at all. That does not make the deployed model safer, but it does mean the benchmark measures something slightly different from what its name implies.

What can actually be proven

Everything above is empirical robustness: the model resisted the attacks that were run. Certified robustness is the stronger claim, that no perturbation inside a radius can change the prediction. Certified accuracy is correspondingly lower.

Jeremy Cohen, Elan Rosenfeld and Zico Kolter’s randomized smoothing gave 49% certified top-1 accuracy on ImageNet against L2 perturbations of norm below 0.5, and was the first certified approach shown to work at full ImageNet resolution.

Deterministic certification at L∞ 8/255 on CIFAR-10 is where the cost shows plainly. The IBP-style certified-training line runs in the mid-thirties: De Palma and colleagues’ CC-IBP reaches 35.27% certified at 53.71% clean, roughly forty points below the 94.78% of an undefended WideResNet-28-10. The deterministic record is higher, and it took the same trick that lifted empirical robustness. Thomas Altstidl and colleagues combined a 1-Lipschitz SortNet architecture with diffusion-generated data at NeurIPS 2024 and certified 41.78%, an improvement of 1.39 points on the previous best. α,β-CROWN has won the international neural network verification competition every year from 2021 through 2025. The verifiers are not the constraint; the models are what will not certify.

For the same dataset and the same radius, 41.78% guaranteed against 73.71% measured. When a vendor uses the word provable, the two questions are the radius and the dataset.

Language has no epsilon ball

Robust optimization needs a perturbation set, a distance, and a projection back into the set after each gradient step. Text supplies no usable version of any of them. Distances over text exist, edit distance among them, but none is simultaneously continuous, semantically faithful, and guaranteed to produce a string an attacker could actually type. The inner maximization, which for images is ten cheap gradient steps, becomes a discrete combinatorial search: GCG runs 500 steps with 512 candidate substitutions per step to optimize a twenty-token suffix. Neel Jain and colleagues noted in their study of baseline defences that crafting a single attack string can take hours on multiple GPUs, which puts a true inner loop out of reach for anything at pretraining scale.

The field has responded with three substitutions, each of which trades fidelity for tractability.

Continuous embedding perturbation. Perturb the embeddings instead of the tokens, recovering a differentiable ball. The perturbations found are not reachable by an attacker who can only supply text.

Latent adversarial training. Perturb hidden activations at intermediate layers. Abhay Sheshadri and colleagues reported that targeted LAT beat the R2D2 baseline on jailbreak robustness with orders of magnitude less compute, and also removed backdoors without knowing the trigger. The threat model is a latent-space ball, which is one further step from anything an adversary controls.

Dynamic attack pools. R2D2, from the HarmBench team, fine-tunes against a pool of test cases continually regenerated by GCG. This is the closest thing in the LLM literature to a genuine inner maximization, and it is expensive for exactly that reason.

The compute problem does not go away with the shortcuts. One July 2026 preprint estimates that the eight-step PGD inner attack alone accounts for roughly 80% of the computation in latent adversarial training, under the Kaplan FLOPs accounting. Robust optimization is expensive because searching for the worst case is expensive, and no reformulation has changed that.

Circuit Breakers is the cautionary case for the whole family. Andy Zou and colleagues published Circuit Breakers at NeurIPS 2024 as a representation-level intervention that interrupts harmful generations rather than filtering them. A year later, an RL-based adaptive attacker reached 100% attack success against it on HarmBench.

The best deployed defence is not a robust model

Anthropic’s Constitutional Classifiers, released in January 2025, survived more than 3,000 hours of red teaming without a universal jailbreak, at a cost of 0.38% additional refusals on production traffic and 23.7% inference overhead. Anthropic’s own January 2026 write-up says the system came close rather than clear: a subsequent public bug bounty did find one universal jailbreak, and later adversarial testing exposed two working families, reconstruction attacks that split harmful content into benign-looking fragments and reassemble it, and output obfuscation that has the model rename reagents as food flavourings. The redesigned system replaces the separate input and output classifiers with one that reads both sides of an exchange, and screens everything through a cheap probe on Claude’s internal activations first, escalating only what that probe flags. Over 1,700 red-teaming hours across 198,000 attempts it produced one high-risk vulnerability, 0.005 per thousand queries, at a 0.05% refusal rate and roughly 1% compute overhead against the 23.7% of the first design. No universal jailbreak so far.

Those are among the strongest published deployment numbers for jailbreak resistance, and the defence is a set of classifiers wrapped around the model. Nothing about the underlying model was made robust. A classifier layer is a system-level control, with its own owners and its own failure modes, and a model-robustness claim says nothing about it. I set that distinction out in secure, safe, responsible and trustworthy AI.

Filters are also models, and Nasr’s group broke a rack of them: above 90% attack success against Protect AI’s detector, PromptGuard and Google’s Model Armor, with PIGuard the most resistant at 71%. They also found that the model-based filters substantially degraded task utility on AgentDojo.

Anthropic and Nasr’s group measured different quantities, and both numbers stand. Anthropic’s bar is a universal jailbreak, one strategy that extracts detailed answers to a whole list of forbidden queries at a level comparable to an unguarded model. Nasr’s bar is per-scenario attack success against a specific target. A system can hold the first line and lose the second one repeatedly. When a vendor quotes a jailbreak resistance number, the useful question is which of those two things they measured, and against an attacker with what budget.

Where this leaves agents

Nasr’s team excluded plan-then-execute defences from their evaluation, and the reason is instructive for anyone building agents. In CaMeL and similar designs, the control flow is determined before untrusted data is read, so an injected string cannot redirect it and the attack fails by construction rather than by strength. CaMeL’s authors report solving 77% of AgentDojo tasks under attack against 84% undefended. Progent cuts indirect injection success on AgentDojo from 39.9% to 1.0%, and on the Agent Security Bench from 70.3% to 3.9%, by enforcing least privilege on every tool call.

Those are static-benchmark numbers, which is the objection this article has spent 2,000 words making. So the interesting result of 2026 is that somebody finally checked. A June 2026 preprint, Adaptive Evaluation of Out-of-Band Defenses, points out that every one of these systems has been validated on a fixed attack set. Its authors specify the protocol an adaptive evaluation requires, then run it against Progent on an open-weight Qwen2.5-7B agent. Progent held. Attack success fell from 25.8% to 4.2%, and a hand-crafted adaptive attack pushed it no higher, to 2.6%. The authors are careful about what that is worth: one small-scale run, a weak model, a single black-box attack template, and an optimized white-box attack still untried.

One result is not a track record. But it is the first time in this literature that an adaptive attacker met a defence and the near-zero benchmark number did not turn into ninety-plus.

The structural point holds regardless. Robust optimization hardens a model against inputs. It does not constrain what the model is permitted to do once it has been persuaded. For an agent holding credentials and tools, the blast radius is set by the permission model, not by how hard the model was to talk into it.

What to ask on Monday

If someone is selling you an adversarially trained or robust model, six questions separate the claim from the evidence.

1. What was the inner maximization? An optimizer running inside the training loop, or a dataset of attack strings? Only the first is robust optimization.
2. What is the perturbation set, written down? Norm, radius, and what falls outside it. A robustness claim without a threat model is not a claim.
3. What was the attacker’s access? MetaSecAlign reported 2% attack success on a static benchmark and 96% against an adaptive attacker on the same benchmark. Static, adaptive and white-box results are not interchangeable. Ask which one produced the number and how many queries the attacker was allowed.
4. Has anyone attacked it adaptively? Nasr’s group argues that a near-zero attack success rate on a public benchmark is itself a warning sign, because most of the twelve defences they broke had reported such a rate before the adaptive attacks ran.
5. What did it cost in clean accuracy and in latency? Somewhere between one and eleven points of accuracy, plus training compute in multiples and, for a classifier layer, inference overhead. If the business cannot pay that, the control is not available at any price.
6. At which layer does the control apply? Model, system, or agent. A model-level result does not hold for the deployed system, and neither one holds for an agent with tool access. That distinction is the one people get wrong most often, and I set it out in AI security 101.

The honest summary

Robust optimization works. It is the only model-level defence in this field that has worked repeatedly, under attack by people trying hard to break it, since 2018. It also delivers a model that gets 73.71% right on a ten-class toy problem when the attacker is confined to one shape of perturbation the defender chose, at a compute cost that made a national laboratory publish scaling laws about it.

Defensive practice has not kept up with that research. In 2018 the community learned that training on a fixed set of attacks produces a model that resists that set and nothing else. In 2025 twelve published defences for language models fell to that same lesson, most of them reporting near-zero attack success right up until somebody tried. The mathematics has not moved much. What has moved is the number of systems now holding credentials, calling tools, and reading untrusted text, defended by techniques whose threat model has never been written down.