Adversarial Attacks on AI: What Has Actually Been Demonstrated

Correction note, 6 September 2026: this article was completely updated and rewritten.

---

In 2017, Kevin Eykholt, Ivan Evtimov and colleagues put black and white stickers on a stop sign and got a neural network to read it as a 45 mph speed limit. The paper, Robust Physical-World Attacks on Deep Learning Visual Classification, went to CVPR 2018 and became the image that has represented AI security ever since. It is on conference slides, in vendor decks, and in the opening paragraph of a great many articles, including the earlier version of this one.

The attack targeted a classifier. The pipeline crops the image to the sign, resizes the crop, and hands it to a classification model. Within months, Jiajun Lu, Hussein Sibai, Evan Fabry and David Forsyth at Illinois ran the same perturbed signs past YOLO and Faster R-CNN in standard configuration and reported that the detectors were not fooled. Their title was “Standard detectors aren’t (currently) fooled by physical adversarial stop signs.” A car does not classify a pre-cropped sign. It detects objects in a scene.

The perturbation defeats the model that was attacked and does not defeat a detector nobody had attacked yet. Attacks on detectors arrived later, from Eykholt’s group at USENIX WOOT 2018, from ShapeShifter against Faster R-CNN, and eventually from Lu and colleagues themselves. Eight years on, the headline still circulates without the detail that the perturbation was tested against a classifier.

Adversarial examples are a real, unsolved property of trained models, and almost every famous demonstration of one defeating a deployed system is weaker than its headline. Holding both halves is the job. Drop the first and you get “adversarial examples are academic,” which is wrong and is contradicted by every defence that has been broken since 2014. Drop the second and you buy products against a threat model nobody has measured.

What an adversarial example is, and what it is not

An adversarial example is an input crafted at inference time so that a model produces an output the attacker chose, or a wrong output the attacker will accept, while satisfying whatever validity constraint the threat model imposes. In image research that constraint is usually a norm bound on pixel change, which is a proxy for human imperceptibility rather than a definition of it. It is not the only option: semantic and unrestricted attacks hold the meaning of the input fixed and abandon the norm bound entirely.

Three mechanisms travel under the word adversarial, and each of them needs a different defence:

• Evasion, the subject of this article. The model is fixed. The attacker perturbs the input at inference.
• Data poisoning, where the attacker corrupts training data so the model learns something wrong. Different lifecycle stage, different access, different controls.
• Prompt injection, where untrusted content the model reads becomes instruction it follows. Not a perturbation and not a norm-bounded problem at all.

NIST AI 100-2e2025, the March 2025 edition of the adversarial machine learning taxonomy, sorts these into evasion, poisoning and privacy attacks for predictive systems, with a separate generative AI section added in that edition. It is the reference to reach for when a vendor’s categories and yours disagree.

The classic demonstration is still the cleanest. Goodfellow, Shlens and Szegedy added a perturbation invisible at normal viewing distance to a photograph of a panda and moved the classifier to “gibbon” at 99.3% confidence. That was Explaining and Harnessing Adversarial Examples in 2014, and nothing since has removed the underlying property.

Why they exist is still contested

Goodfellow’s 2014 paper proposed that neural networks behave close to linearly in high-dimensional input spaces, so small changes to many input coordinates accumulate into a large change in the output. It is the original explanation and it is not a complete one: if linearity were the whole story, single-step attacks such as FGSM should be roughly as effective as multi-step ones, and iterative attacks are reliably stronger. That does not dispose of the underlying insight that near-linear behaviour in high dimensions contributes to the vulnerability.

Tanay and Griffin argued in 2016 for a geometric account instead, in which decision boundaries tilt relative to the data manifold and lie close to the data along directions of low variance.

In 2019, Andrew Ilyas and colleagues at MIT published Adversarial Examples Are Not Bugs, They Are Features at NeurIPS. Their claim is that models learn genuinely predictive features humans cannot perceive, and that adversarial perturbations manipulate exactly those. They supported it by building a training set of mislabelled adversarial images, training on it, and getting a classifier that generalised to the real test set. The same hypothesis accounts for transferability: two models trained on the same distribution learn similar non-robust features.

That explanation is widely cited and it is not settled. Adversarial Examples Are Not Real Features at NeurIPS 2023 found that non-robust features lose their usefulness when carried into contrastive learning, masked image modelling and diffusion models, which is not what a genuine feature of the data distribution should do. The Distill discussion that followed the original paper made a different objection: the hypothesis is a special case of a principle the distributional robustness literature already had, which is that models latch onto superficial statistics.

There is no consensus mechanism. A defence sold on the strength of one of these explanations is a bet on an open research question, and that is worth knowing before the purchase order.

Reading the two demonstrations everyone cites

The stop sign is one. Here is the other, treated the same way.

In February 2020, Steve Povolny and Shivangee Trivedi at McAfee extended the middle bar of the “3” on a 35 mph speed limit sign with about two inches of black tape. The camera read 85, and the car’s Traffic-Aware Cruise Control accelerated. It is a genuine physical-world attack that took eighteen months of work, and it was disclosed to both vendors ninety days ahead of publication.

Four things the headline dropped:

• The cars were a 2016 Model S and a 2016 Model X running the Mobileye EyeQ3, Tesla hardware pack 1. Tesla stopped using Mobileye cameras in 2016. McAfee’s own write-up notes that as of 2020 the newest Teslas did not use the technology and did not appear to support traffic sign recognition at all. That is an assessment of the 2020 fleet, not of what Tesla ships now.
• Mobileye disputed the framing, arguing that the modified sign would have misled a human driver too. That is a real objection to calling it an adversarial attack rather than a legibility failure, and readers should weigh it themselves.
• Neither vendor indicated plans to address it on the existing platform.
• It ran under researcher control, not on public roads.

So the most-cited real-world adversarial attack on a production vehicle concerns a camera generation that was already four years out of date when the work published, on a feature the manufacturer had removed. That is worth saying plainly, and it does not make the research bad. Povolny and Trivedi said explicitly that the point was to get ahead of the curve rather than to discredit anyone. The failure is in the retelling.

The discipline that produces these corrections is not mine and it is not new. In Obfuscated Gradients Give a False Sense of Security at ICML 2018, Athalye, Carlini and Wagner took the nine non-certified defences accepted at ICLR 2018 that claimed robustness, found seven relying on gradient masking, and circumvented six of those completely and one partially. Defensive distillation, which the earlier version of this article recommended, had already been broken by Carlini and Wagner in 2017. It should not appear on a list of current defences and it does not appear on one here.

Where the evidence is strong

Against all of the above, four things are well established and a defender should treat them as settled.

The property is real and has not been removed. Twelve years after Goodfellow’s paper, the top entry on the RobustBench CIFAR-10 leaderboard survives 73.71% of attacks under AutoAttack at a perturbation budget of 8/255. That same model scores 93.68% on clean inputs, while undefended models on CIFAR-10 reach around 99.5%. This is ten classes of 32×32 images, a small benchmark by any standard. The adversarial training article covers the leaderboard and its scaling limits in detail.

Defences break, repeatedly, and usually when someone attacks them properly. That is the ICML 2018 result, repeated at NeurIPS 2020 by Tramèr, Carlini, Brendel and Madry against thirteen further defences, and once more in The Attacker Moves Second, now a USENIX Security 2026 paper, in which a fourteen-author team across Anthropic, Google DeepMind, OpenAI, ETH Zürich and Northeastern broke twelve published jailbreak and prompt-injection defences and pushed most past 90% attack success. A robustness number produced by the defence’s own authors, against attacks they chose, is not sufficient evidence of robustness.

Perturbations transfer. An attacker who trains a surrogate on the same kind of data can often move an attack onto a target they never touched. Transfer is the reason “they do not have our weights” is a weak defence, and it is one of the main routes to a practical black-box attack, alongside query-based gradient estimation and decision-based methods. The white-box versus black-box article works through what each access class actually buys in 2026. Neither label predicts attack strength on its own any more.

The attacks are usually cheap relative to the defences. For standard norm-bounded image attacks, generating adversarial examples is an optimisation problem solved in seconds; physical, semantic and problem-space attacks cost considerably more. Robust optimization costs substantially more training compute and, on most recipes, several points of clean accuracy, and it buys robustness only inside the perturbation set you specified in advance.

Model, system, agent

A finding about a model does not automatically hold for the system built around it, and a system with tools and credentials is different again. The stop sign result is the canonical illustration: classifier yes, detector no, at least at first.

For a defender the practical version is a sequence of questions. Does the attacker reach the model directly, or through a pipeline that crops, resizes, compresses and validates first? Every one of those stages can destroy a perturbation or preserve it. Which one happens is a fact about your pipeline that you have to measure, not a property of the attack. Does a wrong model output become a decision, and is there a check between them? Does the system act, and with whose credentials?

That last question is where the field has moved. An image classifier that misfires produces a wrong label. An agent that misfires calls a tool. The OWASP Top 10 for Agentic Applications, published 9 December 2025 with identifiers ASI01 to ASI10, exists because that difference turned out to need its own taxonomy alongside the LLM Top 10, whose 2026 edition published in early August 2026.

What changed when the field moved to text

Two things, and both weaken the analogy to image classification.

The input space is discrete. There is no small perturbation of a sentence in the sense that there is a small perturbation of a photograph, so the norm ball that made the image threat model writable has no equivalent. What replaces it is a semantic constraint: keep the meaning, change the wording. The semantic adversarial attacks article covers that class specifically.

And the threat model stopped being conveniently expressible as a compact perturbation set. For an image classifier you can state that set in advance and train against it. For an LLM application the set of inputs that cause harmful behaviour is not enumerable ahead of time, which is why defences built on the image analogy keep failing against attackers who adapt.

What to do on Monday

Four things, in order of how much they change.

Ask what was attacked. Model or system, classifier or detector, and through which pipeline. If a vendor cannot answer that about their own demonstration, the demonstration is not evidence about your deployment.

Ask who ran the attack. A defence evaluated only by its authors against attacks they selected has a poor record. Ask whether anyone adapted an attack to the specific defence, and what happened.

Ask for the conditions on every number. An attack success rate means nothing without the model, the access level, the query budget and the defence it ran against. This applies to defensive numbers too: 95% robust accuracy at what epsilon, under which attack, on what dataset.

Then decide whether it matters here. Adversarial robustness is worth paying for where a wrong model output becomes a consequential action with no human or deterministic check between the two, and where an adversary has a reason to want that specific wrong output. Content moderation, fraud scoring, malware classification and biometric access all qualify. A recommendation engine mostly does not. Applied uniformly, robust training pays its clean-accuracy cost on every model in the estate, including the ones an adversary has no reason to attack.

Twelve years of research has left the practical position where it started. Adversarial examples remain a property of how these models learn, and no defence has removed them. The strongest defence available buys bounded robustness at real cost, and only inside a threat model you write down in advance. The demonstrations that circulate fastest are the ones simplified past the point of being true.