Adversarial Training: What It Buys, What It Costs, and Where It Stops Working

The top entry on the RobustBench CIFAR-10 leaderboard survives 73.71% of attacks under AutoAttack at a perturbation budget of 8/255, and it needed a large volume of generated training data to get there. Frontier clean accuracy on the same dataset is about 99.5%. No entry has beaten that number since Brian Bartoldson and colleagues at Lawrence Livermore reported it in 2024, and their scaling laws predict a ceiling near 90% that more compute will not reach.

CIFAR-10 is ten classes of 32×32 images. It is the easiest interesting problem in computer vision, and it has been the field’s benchmark for adversarial robustness for eight years. It is not solved.

Adversarial training is the dominant empirically validated defence against adversarial examples, and it buys robustness inside the threat model you wrote down. Outside that model, robustness holds only where somebody has run the attack and measured it. For an image classifier the threat model is writable, and the price is roughly an order of magnitude more training compute plus five to eight points of clean accuracy. For an LLM application or an agent the threat model is not writable in the same way, and the defences built on the analogy have a short record: the strongest published ones have been broken by adaptive attackers within months of release, repeatedly.

That is a narrower claim than the ones on vendor slides, and a much wider one than “adversarial examples are academic”. Both are wrong. The overclaim sells robustness against a threat model nobody wrote down; the dismissal discards the one defence that survived AutoAttack.

The minimax problem

Christian Szegedy and co-authors found the phenomenon in 2013: a neural network that classifies an image correctly can be made to classify it as something else by a change too small for a person to see. Ian Goodfellow, Jonathon Shlens and Szegedy followed in 2014 with the fast gradient sign method, which produces such a change in a single backward pass, and with the observation that training on those examples improves resistance to them.

The formulation that the field still uses came from Aleksander Mądry’s group at MIT in 2017. They wrote robust training as a saddle-point problem: minimise, over model parameters, the expected value of the maximum loss an adversary can produce inside a perturbation set around each training point. Two optimisations nested inside each other. The inner maximisation constructs the worst input the adversary can build within the budget, usually with projected gradient descent. The outer minimisation trains the model on that input instead of the clean one.

Adversarial training is not data augmentation with noise. Augmentation samples perturbations. Adversarial training searches for the worst one, at every step, for every example in the batch. That search is what makes the method effective and it is what makes it expensive.

It also means somebody chooses the perturbation set before training starts. That choice fixes what the word robust will cover. The perturbation set is almost always an Lp ball: all changes whose maximum per-pixel shift is under some ε (the L∞ ball), or whose Euclidean length is under some ε (the L2 ball). Choosing 8/255 in L∞ on CIFAR-10 is a convention, not a measurement of what an attacker can do. A model trained against that set is robust to that set.

The attacker’s side of these mechanics is already here: gradient-based attacks for the inner maximisation, perturbation attacks for the geometry of the budget, and model evasion for the case where the attacker wants one specific wrong answer instead of any wrong answer.

Everything else got broken

Adversarial training dominates the literature because the alternatives broke under examination.

In 2018 Anish Athalye, Nicholas Carlini and David Wagner looked at the nine non-certified white-box defences accepted at ICLR that year, found that seven of them relied on a shared failure mode they called obfuscated gradients, and broke six of those completely and a seventh partially. The defences had made the gradients uninformative instead of making the models robust. A gradient attack against such a model fails to find the perturbation that exists, and the benchmark records that failure as robustness.

That paper set the evaluation standard the field now runs on. Francesco Croce and Matthias Hein’s AutoAttack, published in 2020, bundles four attacks into a parameter-free ensemble: three white-box gradient attacks and Square, which is a black-box query attack. The Square component is what catches gradient masking, because a model that survives the gradient attacks only by making its gradients useless has no answer to an attack that never computes one. RobustBench then standardised the leaderboard around it, and the effect on the literature was immediate: reported robustness numbers fell, and the entries that survived were overwhelmingly adversarial training and its variants.

NIST reached the same place in its taxonomy. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, published 24 March 2025 as NIST AI 100-2e2025, treats adversarial training as one of the principal mitigations studied for evasion against predictive systems, and spends as much space on the limitations of widely used mitigations as on the mitigations themselves. Presenting a control alongside its limits is the right framing, and it is not how the technique is usually sold.

There is a lesson in the 2018 episode that is worth carrying into 2026, because it keeps recurring. Seven defences reported strong numbers and passed peer review at ICLR. They were wrong, because nobody had run an adaptive evaluation. The attacker in the paper was not trying to break that specific defence. Every claim in the sections below should be read with that in mind, including the claims about adversarial training itself.

The dial nobody mentions

“Adversarial training” names a family, not a method, and the members differ in ways that change what you get. Anyone commissioning this work should know at least four of them, because the choice between them is a product decision disguised as a hyperparameter.

PGD adversarial training, the Mądry recipe, trains only on the adversarial example. It is the baseline everything else is measured against and it is the most expensive of the standard options.

TRADES, from Hongyang Zhang and colleagues in 2019, decomposes robust error into two parts: the natural error the model would make anyway, and the boundary error introduced by the adversary being able to push examples across the decision boundary. It then trains on both terms with an explicit weighting between them. The consequence is the useful part. TRADES gives you a knob. Turn it one way and you keep more clean accuracy and lose robustness; turn it the other way and you trade in the opposite direction. Most of the frontier entries on the leaderboard descend from this formulation, and what it adds over the Mądry recipe is that you set the trade-off yourself instead of accepting the one the recipe produces.

For someone running a production model, that knob means the seven-to-eight-point accuracy cost quoted earlier is not a fixed price. It is the price at the setting the papers chose, which is the setting that maximises a leaderboard number. Your product may want a different point on that curve, and you can have it.

MART and adversarial weight perturbation are the two refinements worth knowing by name. Yisen Wang and co-authors showed that misclassified training examples contribute differently to robustness than correctly classified ones and that treating them differently helps. Dongxian Wu, Shu-Tao Xia and Yisen Wang showed that flattening the loss landscape with respect to the weights, not just the inputs, closes part of the robust generalisation gap. Both appear inside most current frontier recipes.

Generated-data adversarial training is not a different algorithm, it is the same algorithms with a diffusion model producing training examples. It has been the largest single driver of leaderboard movement between 2021 and 2024, including the current top CIFAR-10 entry. If a paper reports a number substantially above the field and does not say whether it used extra data, that omission is the first thing to check.

What the numbers actually are

In July 2026, Niklas Bunzel at Fraunhofer SIT posted VanillaBench, a preprint that does something the robustness literature had mostly not done: it compares every adversarially trained model on RobustBench against the clean accuracy a normal model of the same architecture achieves. The result is the clearest available picture of what this technique costs, and it draws on 186 robust models across four tracks.

On CIFAR-10 at L∞ ε = 8/255, across 98 robust models published between 2018 and 2024, the mean clean accuracy is 87.49% and the mean robust accuracy under AutoAttack is 53.44%.

On CIFAR-100 at the same budget, across 38 models, mean clean accuracy is 66.61% and mean robust accuracy is 32.12%. A harder classification problem costs about 21 points of robust accuracy, a drop of roughly 40% relative to CIFAR-10.

On ImageNet at L∞ ε = 4/255, across 30 models, mean clean accuracy is 73.12% and mean robust accuracy is 46.69%.

The clean-accuracy cost, measured against a normal model of the same architecture rather than against the best model on the leaderboard, averages 7.59 points on CIFAR-10 L∞, 8.52 points on ImageNet, 17.47 points on CIFAR-100, and 3.50 points on CIFAR-10 at L2 ε = 0.5. Bunzel found only two cases in the whole set where an adversarially trained model beat its architecture-matched clean baseline, both by less than half a point.

Those are the aggregates; the frontier reaches 73.71% under AutoAttack and no further. Bartoldson’s group reported two separate results that are often merged: they matched the previous state of the art using 20% fewer training FLOPs and 70% fewer at inference, and, in a larger run, reached 73.71% AutoAttack accuracy for a three-point gain. Their scaling laws then predicted that robustness on this problem grows slowly and plateaus near 90%, so reaching that ceiling by scaling is impractical. The part of the paper people skip is the human study: they showed people the images that successfully attacked their best model, and human performance also topped out near 90%, because L∞ perturbations at that budget produce images that no longer match their original label. Part of the apparent gap is therefore a label-validity problem created by the threat model, not a failure of the model.

Five conditions apply to every one of those numbers:

• Dataset. CIFAR-10 robustness does not predict ImageNet robustness, and neither predicts robustness on your data.
• Norm and budget. L∞ at 8/255, L2 at 0.5, L∞ at 4/255 are three different problems with three different answers.
• Attack and query budget. AutoAttack is a standard, not a bound. A stronger attack exists until someone tries.
• Access. A white-box adversary knows the weights and the defence, so it is the stronger capability set and its results are a lower bound on robustness. The opposite error matters more. When an attack silently fails or was never adapted to the defence, the evaluation overestimates robustness.
• Extra data. Leaderboard entries trained with generated or external data are not comparable to entries that were not, and the difference is worth several points.

When a vendor tells you their model is robust, the useful reply is: robust to what ball, at what radius, under what attack, with what access, evaluated by whom. Four of those five questions have a number for an answer.

The costs that do not appear in abstracts

Compute. The inner maximisation is not free. PGD with k steps means k additional forward and backward passes per batch. The common recipes use seven to ten steps, so adversarial training costs roughly an order of magnitude more than standard training for the same number of epochs. That is the single fact that decides whether this is on the table for a given team.

Two lines of work attacked that cost. Ali Shafahi and colleagues recycled gradient computations to get robustness “for free” in 2019. Eric Wong, Leslie Rice and Zico Kolter showed in 2020 that single-step FGSM training with random initialisation could match PGD training at a fraction of the cost. Both work, and both introduced a failure mode called catastrophic overfitting: partway through training, accuracy against the single-step attack shoots toward 100% while accuracy against a multi-step attack collapses to zero, sometimes within a couple of epochs. The model has learned to defeat the specific attack used in training rather than to be robust. It is the obfuscated-gradients problem reappearing inside the training loop, and it is why fast adversarial training needs monitoring, regularisation such as GradAlign, or an early-stopping trigger. Fast adversarial training is cheap only while the monitoring catches catastrophic overfitting.

Robust overfitting. Rice, Wong and Kolter showed in 2020 that robust test accuracy peaks early in adversarially trained models and then degrades for the rest of training, even while robust training accuracy keeps improving. Early stopping on a robust validation set recovers most of the gap and beats nearly every algorithmic improvement published up to that point. Running this pipeline requires a held-out adversarial validation set. Stop training at the peak, not at the end.

Data. Ludwig Schmidt and colleagues argued in 2018 that robust generalisation needs substantially more data than standard generalisation, and the empirical work since has agreed. The leaderboard moved when Sven Gowal’s group at DeepMind started adding generated data in 2021 and again when Zekai Wang and co-authors improved the generator with diffusion models in 2023. Without a way to synthesise millions of in-distribution examples, you are training against a harder version of the problem than the papers were.

Accuracy. The trade-off is real, Dimitris Tsipras and colleagues argued in 2018 that it may be fundamental rather than an artefact of current methods, and VanillaBench puts a number on what it costs today. Seven to eight points on the tasks where the method works best. On a fraud model or a medical triage model, seven points of clean accuracy is not an abstraction, it is a caseload.

Compute, early stopping, a synthetic data pipeline and seven points of clean accuracy are what the control costs, which is a normal price list for a security control. The failure is presenting it as a fix.

Robustness outside the trained threat model has to be measured

This is the limitation that determines whether the method is useful for your problem, and it is the one most often left out.

A model adversarially trained on L∞ perturbations is robust to L∞ perturbations. Its robustness to L2 or L1 perturbations is much weaker, and the effect has been documented repeatedly since Florian Tramèr and Dan Boneh’s work on multiple perturbations in 2019. Training for several norms at once used to cost a multiple of a single-norm run, until Croce and Hein showed in 2022 that a short fine-tune on the extreme norms gets multi-norm robustness for roughly the price of one.

Norms are the easy case, because at least they are comparable. The harder case is perturbations that are not norm-bounded at all: spatial transformations, colour shifts, image filtering, and on-manifold perturbations that stay inside the space of realistic images. Sihui Dai, Saeed Mahloujifar and Prateek Mittal formalised the mismatch in 2022, relating the gap between the threat model you trained against and the one you face to the variation of the learned feature extractor.

The evidence here does not support the stronger claim that robustness never transfers, and the counter-example is worth stating because it cuts against the argument I am making. Cassidy Laidlaw, Sahil Singla and Soheil Feizi built a neural perceptual threat model precisely because Lp balls describe badly what a person would call “the same image”. Perceptual adversarial training then generalised across several attack families it had never been trained against, beating models trained on multiple Lp attacks directly. Transfer happens. What does not happen is transfer you can assume without measuring it, and the same paper supplies that half too: the multi-attack-trained baselines failed on the perceptual attacks.

So the practical consequence is narrower and more useful than “it does not generalise”. Adversarial training converts an open-ended security question into a closed-form optimisation problem by requiring you to specify the adversary in advance, as a set. Performance against an adversary outside that set is unestablished until somebody runs the attack. Sometimes it is fine. Sometimes a printed sticker on a stop sign defeats a model that scores well on L∞ benchmarks, which is why the physical-world attacks discussed in risks from AI autonomous robots get treated as their own problem rather than a special case.

The question to ask before spending anything on this: can I state what my attacker is allowed to do precisely enough to optimise against it, and then check the result against an attack I did not train on? If yes, adversarial training is on the table. If the honest answer is that nobody can say what the adversary is allowed to do, the number a vendor quotes you is measuring their imagination.

Where it earns its keep right now

One of the clearest deployable applications of adversarial training is the vision encoder inside a multimodal system.

CLIP and its variants are used as frozen vision encoders in a large number of vision-language models, LLaVA and OpenFlamingo among them. An off-the-shelf CLIP encoder has, in the comparison table of Christian Schlarmann, Naman Deep Singh, Croce and Hein’s ICML 2024 paper, 0.0% robust accuracy on ImageNet at L∞ ε = 1/255, and 0.0% again at 2/255 and 4/255. Not degraded. Effectively zero, at a budget an eighth of the CIFAR-10 convention.

Their method, FARE, adversarially fine-tunes the vision embedding without labels, for two epochs on ImageNet, by minimising the distance between the perturbed embedding and the original model’s clean embedding. The predecessor technique, TeCoA from Chengzhi Mao and colleagues, did the same thing in a supervised way against a zero-shot classifier and produced a robust encoder whose downstream captioning quality was noticeably worse. FARE gives up some ImageNet robustness and keeps much more of the original model’s general-purpose behaviour, and it drops into a downstream vision-language model with no retraining of the downstream model at all.

FARE has four properties: a bounded perturbation in a genuinely continuous input space, a swappable component, an adaptive-attack evaluation, and a stated cost. It is also still small, and the number depends on which checkpoint you take. FARE is trained at a chosen radius and the released models differ. For the encoder fine-tuned at 1/255, robustness is meaningful at 1/255, weaker at 2/255 and essentially gone by 4/255. The honest description is that it removes the trivial attack rather than the determined one. For a system that ingests images from the open web, removing the trivial attack is worth doing. It is not the same as being secure, and the paper does not claim it is.

Anyone building on visual inputs should read that result alongside multimodal attacks, which covers the attack side of the same surface.

Language models break the assumption

Everything above requires a continuous input space with a distance function that tracks what a person would call the same input. Text has the second problem badly. Tokens are discrete, and no canonical metric on prompts corresponds to Lp on pixels. The changes an attacker makes are semantic where Lp perturbations are geometric. An attacker rephrases, roleplays, translates, encodes, or appends an optimised suffix. No Lp ball contains a rephrasing.

The field responded by relaxing the problem in three directions, and the results are genuinely mixed.

Train on the attacks themselves. Mantas Mazeika and colleagues’ R2D2, released with HarmBench in 2024, ran adversarial training against a continuously updated pool of optimised jailbreak strings. It works and it is expensive, and it inherits the generalisation problem in a new form: robustness to the attack family you trained on.

Relax to a continuous space. Sophie Xhonneux and co-authors attacked continuous embeddings rather than discrete tokens during training, which restores the gradient-based inner maximisation at far lower cost than discrete search. Their models transferred to discrete attacks they were never trained on. As with the perceptual result above, the relaxation you optimise against need not match the attack you face. Whether it does in a given case is something somebody measured.

Attack the latent space. Stephen Casper and colleagues and then Abhay Sheshadri and colleagues perturbed hidden activations rather than inputs, on the argument that safety fine-tuning suppresses harmful capabilities rather than removing them, and that a latent adversary reaches the suppressed behaviour more directly than an input adversary can. Targeted latent adversarial training beat the R2D2 baseline with orders of magnitude less compute, and also removed backdoors without knowing the trigger.

One result from this literature is unusually actionable. Shaopeng Fu and co-authors found that the jailbreak attack success rate against an adversarially trained model correlates with the ratio of the square root of the test-time adversarial suffix length to the suffix length used during training. Training against short suffixes buys meaningful protection against longer ones, with a stated relationship rather than a hope, which makes the compute trade-off calculable. That is the kind of finding this field produces too rarely.

And then the counter-evidence, which is the part that decides how much weight to put on any of it. Luke Bailey and colleagues showed in late 2024 that activations can be obfuscated: an attacker can find inputs that produce the harmful behaviour while keeping the latent representations outside the region the defence monitors or was trained on. They demonstrated it against both latent adversarial training and circuit breakers. Separately, John Hughes, Sara Price, Aengus Lynch and colleagues at Anthropic broke a circuit-breaker model at a 52% attack success rate by repeatedly resampling the same request with random capitalisation and character shuffling. No gradients, no white-box access, and no optimisation more sophisticated than trying again. The same undefended base model fell at 94%, so the defence did work: it took the attack success rate from 94% to 52%.

Athalye, Carlini and Wagner described this in 2018. A defence trained against one adversary is evaluated against that adversary and reports a good number. A different adversary arrives and the number does not hold.

The prompt injection sequence, in order

The clearest case study in the whole subject is the fine-tuning defence line against prompt injection, and it is worth walking through in order because the sequence is the argument.

StruQ introduced a structured prompt format with reserved delimiters and fine-tuned the model to privilege the instruction field over the data field. SecAlign, from Sizhe Chen and colleagues, extended it with preference optimisation on paired desirable and undesirable outputs. The reported results were strong: optimisation-free attacks reduced to roughly zero, and single-digit to low-teens attack success under GCG across the models tested, against prompting-based defences that failed most of the time. Meta SecAlign scaled the approach to an openly released foundation model in July 2025.

In July 2025, the ASTRA work attacked the defended models directly with an architecture-aware method and reported 82.5% and 72.5% attack success against the SecAlign-defended Mistral and Llama-3 models, obtained from the original authors, at a 35-token budget on a 40-example subset. At SecAlign’s original 20-token suffix-only protocol the same attack is weaker, which is worth stating: part of the collapse is a larger budget, not only a better attack.

In March 2026, Alexander Panfilov and co-authors pointed Claude Code at thirty-odd existing attack implementations and let it iterate on the algorithms themselves. The discovered attacks transferred from surrogate models to held-out ones and reached 100% attack success against Meta-SecAlign-70B, against 56% for the best hand-built baseline. Their argument is the one that matters here: automated adaptive attack should be the minimum bar a defence has to clear before publication.

Also in March 2026, Chenlong Yin, Runpeng Geng, Yanting Wang and Jinyuan Jia published PISmith, which trains an attacker model by reinforcement learning against a defended target in a black-box setting where it can only send queries and read outputs. Against Meta-SecAlign-8B, static attacks achieved small success rates, confirming the original robustness claim. Search-based methods reached single-attempt rates of 11% to 21%. PISmith, trained on 100 samples from one dataset and evaluated across twelve unseen benchmarks, reached 100% attack success within ten attempts and 87% on a single attempt. Among the defences they evaluated, none combined high task utility with low adaptive attack success.

And then, on 21 August 2026, the story turned. Sizhe Chen and colleagues, the same group whose defence PISmith had just dismantled, published SecOPD, which replaces the sequence-level training signal used by preference optimisation with token-level feedback: the defended model produces a response to an injected input, and each token is scored by the original model given the clean input. Their Qwen3.6-27B reaches 9.0% attack success against PISmith, where Meta-SecAlign suffers 94.0%. On AgentDojo, a domain absent from training, it reaches 4.7% against Meta-SecAlign’s 5.5%, with utility within a few points of the undefended model. The paper is accepted to EMNLP 2026.

That last step is what makes the sequence worth reading rather than just depressing. It is a healthy research cycle. Chen’s group published the defence and PISmith broke it. They returned with a token-level training signal built against PISmith. An order-of-magnitude reduction against the strongest published attack is a real result and it deserves to be reported as one.

It is also two weeks old at the time of writing, evaluated by its own authors, and scoped to indirect prompt injection with a known trust boundary. The honest reading of SecOPD is not that prompt injection is solved, and its authors do not claim it is. Their own paper implies the general rule: the unit of progress in this field is not “this defence works”, it is “this defence survived this named adversary on this date”. Milad Nasr, Nicholas Carlini and a large group at Google DeepMind made the general version of that argument in 2025, and the title says it: the attacker moves second.

Every number in that sequence is true. Each defence was robust against the attacks its authors ran, which is what the evaluation measured. Each was then measured against an adversary built to defeat it, and the second number is the one that describes your exposure.

There is a structural point underneath the empirical one. Model susceptibility to injection is a model property, and the results above show it is a movable one. Whether that susceptibility becomes a breach is a system property, and it is decided by what the compromised component is allowed to do. Adversarial training operates on the first and changes nothing about the second. This is the model, system and agent distinction doing real work: an agent holding credentials and tools has a blast radius that a 9% attack success rate does not describe.

Which is why the controls that bound the damage are architectural rather than statistical. Separate the privileged and unprivileged paths. Do not give a component that reads untrusted input the authority to take consequential actions. Require human confirmation on irreversible operations. Constrain which tools are reachable from which context. A line of work from CaMeL onward builds this as a deterministic reference monitor enforcing policy at the point where an action takes effect, which is a design pattern security has used since the 1970s. Those controls are boring, and they hold a property that no amount of adversarial training holds: they do not require you to have enumerated the attacker’s moves in advance.

Certified robustness, and why it is not the answer either

The alternative to measuring robustness is proving it. Jeremy Cohen, Elan Rosenfeld and Kolter’s randomized smoothing gives a probabilistic guarantee: wrap a base classifier in Gaussian noise, take a majority vote over many noisy copies, and derive a radius within which the smoothed classifier’s prediction provably cannot change.

A guarantee is worth more than a benchmark, and there are three things to know before treating it as the answer.

The certified radii are small, and smaller than the empirical robustness adversarial training achieves at the same clean accuracy. The inference cost is high, because certification requires many forward passes per prediction. And the guarantee is still stated over a norm ball, so certification inherits the threat-model problem in full. A certified L2 radius says nothing about a rotation, a colour shift, or a physically printed patch.

Certification proves that no perturbation inside a stated radius changes the prediction, which is a claim about the radius and not about the attacker. Those are only the same statement if the attacker agreed to stay inside your ball.

How to check a robust model yourself

If someone hands you an adversarially trained model, or if your own team produces one, the evaluation is where the value is won or lost. Nicholas Carlini, Anish Athalye, Nicolas Papernot and a large group of co-authors wrote the checklist in 2019 and it has aged well. Four of its checks catch most of the failures, and all four are cheap.

Run an unbounded attack and confirm the model breaks. If you let the perturbation budget grow without limit and robust accuracy does not fall to zero, the attack is failing rather than the defence holding. This one check would have caught most of the 2018 cohort.

Increase ε and confirm robustness falls monotonically. Attack success should never decrease as you give the attacker more room. If robustness stays flat or rises as the budget grows, the attack is failing, not the defence holding. A sharp drop just past the training radius is worth investigating but is not by itself a fault; robustness has no obligation to decay smoothly.

Compare a gradient attack against a gradient-free one. A white-box gradient attack has strictly more information than a black-box query attack and should beat it. If Square, which is inside AutoAttack, outperforms the gradient attacks, your gradients are broken and every white-box number you have is an overestimate. Square outperforming the gradient attacks is the signature of obfuscated gradients.

Attack the defence, not the model. The single most common evaluation failure, in 2018 and in the SecAlign sequence in 2025 and 2026, is running a standard attack against a defended model instead of designing an attack that knows the defence is there. A defence evaluated only against attacks that predate it has been tested against nothing. This is the same reasoning as adaptive attacks against intrusion detection, where a detector tuned to known evasion patterns fails the moment the evasion adapts to the detector.

Two practical notes. AutoAttack on a full ImageNet validation set is expensive, which is why RobustBench evaluates a fixed 5,000-image subset. Use the same convention so your numbers are comparable. And record the query budget. A model that resists 100 queries and falls at 10,000 is a different security proposition from one that survives both, which matters most for the query-based attacks that a black-box adversary against a hosted API can actually run.

What to do about it

Decide whether the technique applies at all. The test is not whether your inputs are continuous, because continuous-embedding and latent-space training have carried adversarial training into language models. The test is whether you can state an adversarial objective precise enough to optimise against, and then validate the result against an attack you did not train on. If the honest description is “changes to a continuous input, bounded in a norm, that a person would not notice”, adversarial training is the right tool and the literature will tell you roughly what you will get. If nobody on the team can say what the adversary is allowed to do, adversarial training is not a control you can rely on, whatever the number on the slide says.

Budget it properly. Roughly an order of magnitude more training compute for a PGD recipe, or a fast recipe plus catastrophic-overfitting monitoring. A held-out adversarial validation set and early stopping, because robust overfitting is not optional to handle. A synthetic data pipeline if you want frontier numbers. Five to eight points of clean accuracy on the tasks where it works best, more on harder ones, and a decision from whoever owns the product about whether that is acceptable.

Take the cheap win where there is one. If you have a vision-language model with a frozen CLIP encoder ingesting untrusted images, replacing that encoder with an adversarially fine-tuned one is two epochs of work, requires no downstream retraining, and moves robust accuracy at small budgets off zero. That is a good trade and very few teams have made it.

Interrogate every robustness claim with five questions. Which dataset. Which norm and which ε. Which attack and what query budget. White-box or black-box, and if black-box, how many queries. Who ran the evaluation, and was it adaptive. A vendor who cannot answer all five has not measured robustness. Without an ε, the word names no radius.

Treat published robustness as perishable, and date it. StruQ, SecAlign, Meta-SecAlign, R2D2, latent adversarial training and circuit breakers were each broken after publication, by someone who read the paper. The corollary runs both ways: SecOPD is currently the strongest published result against adaptive prompt injection and it is weeks old, which means it is promising rather than proven. Record the date and the named adversary next to any robustness number you rely on. Re-evaluate against current attacks on a named cadence, written into the control document. This belongs in the AI security function’s standing work, alongside the responsibilities described in the chief AI security officer role.

Do not let adversarial training substitute for the controls it does not replace. Standard input-space adversarial training defends against test-time evasion and is not a defence against data poisoning or backdoors, which attack the training pipeline. The exception is narrow and named above: targeted latent adversarial training removed backdoors without knowing the trigger, under a different threat model and at the level of activations rather than inputs. It does nothing about model extraction or model inversion, which target confidentiality rather than integrity. It is one control against one class of attack, and the four-way distinction between secure, safe, responsible and trustworthy AI set out in addressing the full stack of AI concerns exists partly to stop this collapse from happening in procurement documents.

What would change the recommendation

Three results would move this, and none of them has arrived.

A method that pushes CIFAR-10 past 80% under AutoAttack without extra data would suggest the plateau in the scaling laws is an artefact of current techniques rather than a property of the problem. Bartoldson’s analysis says not to expect it from scale alone.

A defence that generalises across threat models without being told the threat model would change the argument entirely, because the specification requirement is the binding constraint, not the compute. Perceptual adversarial training is the closest anyone has come and it still requires a threat model, just a better-shaped one.

An adversarial training result for language models that holds against well-resourced adaptive attackers for a couple of years would justify treating it as a control rather than a layer. SecOPD is the first candidate that looks serious enough to be worth watching, and it is a few weeks old, evaluated by its own authors, and scoped to indirect injection with a known trust boundary. Ask again in a year. If PISmith’s successors have not moved its 9% much, this recommendation changes for that specific setting, and only that one.

Until then the position is the one in the third paragraph. Adversarial training works, on a stated threat model, at a price, and its value is exactly as good as the evaluation somebody ran against an attacker who was trying. Defensive practice on everything else lags the offensive research by years, and the useful thing to tell a board is not that the model is robust but which adversary it was tested against, and when.