White-Box vs. Black-Box Attacks: What Access Actually Buys

In October 2025, a fourteen-author team spanning Anthropic, Google DeepMind, OpenAI, ETH Zürich, Northeastern and several smaller labs posted The Attacker Moves Second, which broke twelve published defences against jailbreaks and prompt injection. Most of the twelve had originally reported near-zero attack success rates. The team pushed most of them past 90%. The work has since appeared at USENIX Security 2026.

The authors had four methods available: gradient descent, reinforcement learning, search-based optimisation, and five hundred humans in a red-teaming competition. They ran the gradient attack against exactly one of the twelve defences, RPO, and wrote that gradient-based attacks on text remain unreliable and that they generally recommend attacks operating directly in text space. On RPO, evaluated on HarmBench, gradient optimisation reached 96% and reinforcement learning 98%. Across the twenty-nine scenarios where the automated search attack and the human competition overlapped, the search attack succeeded 69% of the time and the human participants, treated collectively, 100%. The best individual competitor found a break in as few as fifty queries; the search attack needed eight hundred to reach its 69%.

The attackers who produced those numbers had no access to weights, gradients, architecture or internal states. Competitors typed into a web form and watched the agent’s tool calls stream back, and the best individual participant across all twenty-nine scenarios still beat the automated search, 75% to 69%.

For LLM systems, access class alone no longer predicts attack strength. That cuts both ways. “This result assumed white-box access, so it does not apply to us” is not the reassurance it was in 2019. “We tested black-box and it held” is weak evidence on its own. Four things decide whether it means anything: which black-box oracle, whether the attack was adapted to the defence, the query budget, and the feedback the system returned. The question to ask about a security claim is not which of the two labels applies, but what the attacker actually held, and whether your deployment gives a real adversary more than that or less.

What the three settings actually mean

The vocabulary is older than the confusion, and there are two overlapping versions of it. NIST AI 100-2e2025, the March 2025 edition of the adversarial machine learning taxonomy, sorts attacks into three knowledge classes: white-box, black-box, and gray-box for everything in between. The Attacker Moves Second uses a narrower operational split, built for its experiments. That split describes an API:

White-box. The adversary has the model architecture and its parameters, can observe internal states, and can compute gradients of any internal or output value with respect to inputs or parameters. This is the setting behind most of the adversarial attack literature and behind every gradient-based method in it. NIST’s own white-box class is wider again, reaching to the training data and the hyperparameters.

Black-box with scores. The adversary can query the model with arbitrary inputs and receives output scores, logits or probabilities, along with the prediction or generation, but never parameters or gradients.

Black-box, generation only. The adversary queries and sees the final output, text or label. Nothing else.

Both of the second two are black-box in NIST’s sense. Splitting them refines the black-box case operationally within NIST’s three classes, and the third of them matters more than either endpoint. Gray-box covers partial knowledge: architecture but not parameters, parameters but not training data, a similar data distribution rather than the real one. It is not a hedge and it is not rare. Real products usually expose different amounts at different layers, which is why a single system-wide label fails more often than it works.

There is a second question that the labels quietly skip: access to what. A model, a system, and an agent are three different attack surfaces, and access to one says little about the others. Run an unmodified open-weight checkpoint behind your own API and an attacker has white-box access to your model and black-box access to your system. Wrap a frontier API in a defence architecture described in a published paper and the reverse holds: the model is opaque, the system is documented. The Attacker Moves Second measured that second case directly, and I come back to the number below.

Why white-box was the safe assumption

For a decade the field treated white-box as the conservative choice, and it was right to.

The reasoning came from cryptography. Kerckhoffs’s principle, from 1883, says a system should stay secure when everything about it except the key is public. Applied to machine learning, that means you evaluate against an adversary who has the weights and the code. Any weaker assumption is a bet on the attacker’s ignorance, and that bet has a poor record.

Image classification made the assumption cheap to hold. The input space is continuous, the model is differentiable, and projected gradient descent finds minimal perturbations reliably and quickly. White-box was the strictly stronger information model, and in that setting the weights and gradients converted efficiently into working attacks, which is what made evaluating white-box the conservative choice. Black-box robustness was never a claim anyone respectable made, because the obvious follow-up was “and what happens when they get the weights?”

The discipline that grew from that assumption is the reason the field has any evaluation standards at all. In Obfuscated Gradients Give a False Sense of Security at ICML 2018, Athalye, Carlini and Wagner examined the nine non-certified ICLR 2018 defences that claimed white-box robustness, found seven of them relying on gradient masking, and circumvented six completely and one partially. On Adaptive Attacks to Adversarial Example Defenses repeated the exercise at NeurIPS 2020 on thirteen more. A defence evaluated against attacks its authors did not design it to survive tells you nothing, and the intrusion detection work on adaptive attackers had been making that point in a different register for years.

What text changed

Two things broke the hierarchy when the field moved to language models.

The first is discretisation. Gradients are computed in a continuous embedding space and tokens are discrete. Every gradient-based text attack therefore has to project its update back onto real tokens, and that step makes the whole procedure brittle. Greedy Coordinate Gradient, the reference method from Universal and Transferable Adversarial Attacks on Aligned Language Models, takes five hundred optimisation steps evaluating five hundred and twelve candidates each to produce a twenty-token suffix. Those are local evaluations against weights the attacker already holds rather than calls to somebody’s API, which is the only reason the cost is bearable. The suffixes come out as gibberish, which a perplexity filter can catch. Full weight access buys an expensive, brittle, conspicuous attack.

The second is that the input space is natural language, and every fluent adult is already an expert in it. The trigger that beat the Protect AI detector in that study was a fake internal-policy notice: it told the agent that rescheduling a calendar event now required deleting a temporary tracking file, and named the file. A person who has never heard of an embedding can write that sentence. Nothing in it looks like an attack out of context, which is exactly why a classifier cannot flag it without drowning in false positives on ordinary business text.

The numbers from the study track that. On AgentDojo against an undefended Gemini 2.5 Pro, the benchmark’s own static attack template succeeded 30% of the time and the LLM-guided search attack succeeded 100% of the time, with a median of thirteen queries to the first working trigger. MetaSecAlign, whose own paper reports a 2% attack success rate on static AgentDojo, fell to 96% under the same search attack. Spotlighting and prompt sandwiching, at attack success rates as low as 1% against the static set, went above 95%. Circuit Breakers reached 100% under reinforcement learning on HarmBench. Not one of those attacks computed a gradient.

Every deployment is gray-box

The reason the labels mislead defenders is that “black-box” gets read as a property of the system rather than a description of one attacker on one day. Four channels move a real adversary up the scale, and none of them requires anything exotic.

Open weights hand over the model

If your product runs Llama, Qwen, DeepSeek, Mistral or gpt-oss behind your own API, an attacker can download the same weights and never send you a query to get them. Deploy that checkpoint unmodified and it is exact white-box access to your model. Fine-tune it, quantise it or bolt on an adapter and the attacker holds the public base instead, which is an excellent surrogate rather than a copy. Either way the gradient literature, the evasion techniques and the activation-space methods are in play, and the work happens on the attacker’s hardware where you cannot see it, rate-limit it, or log it.

Fine-tuning does not restore secrecy either. Kuo, Yadav and Smith showed in Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks in May 2026 that two safeguards built specifically to survive adversarial fine-tuning, TAR and SEAM, fall to two cheap attacks that use no gradient optimisation at all: abliteration, which computes the model’s refusal direction and projects it permanently out of the attention and MLP output matrices, and prefilling, which injects a compliant opening into the assistant’s turn before generation starts. Across BeaverTails, HarmBench and AdvBench, attack success rates against the safeguarded models went from under 10% to between 16% and 96%, with TAR holding up considerably better than SEAM.

White-box is a bundle of four capabilities: gradient computation, weight editing, activation reading, prefill control. A defence can be robust against one member of the bundle and fail against another. A paper that reports “robust in the white-box setting” has told you which member its authors tested and nothing about the others.

Scores and parameters set the access class

In Stealing Part of a Production Language Model at ICML 2024, Carlini and colleagues recovered the entire embedding projection matrix, up to symmetries, of OpenAI’s ada and babbage models for under twenty dollars, using nothing but the public API. They confirmed the hidden dimensions, 1024 and 2048, that no public source had established. They recovered the hidden dimension of gpt-3.5-turbo and estimated the full matrix at under two thousand dollars in queries.

The mechanism is the interesting part for a defender. The attack worked because a few providers exposed the logit_bias parameter, which, combined with top-k log probabilities, let the researchers reconstruct precise logit values a normal caller never sees. Providers who did not expose that parameter were not vulnerable to it. Nobody made an architectural mistake. Someone made a product decision about API surface, and exposing logit_bias turned a generation-only oracle into a score-rich one, which is the difference between a query attack being infeasible and it costing twenty dollars. Both are still black-box in NIST’s sense, and that is the point: the label did not move and the exposure did. A guardrail that returns a confidence score is an oracle too, and the search attack in the Attacker Moves Second study fed exactly that score back into its loop to reach above 90% against Protect AI, PromptGuard and Model Armor.

Audit your own API surface for this. Log probabilities and confidence scores are score channels, verbose errors leak internal state, response timing is a side channel, and a detector verdict is a pass-fail oracle. They differ in kind and not in consequence, because each one gives an optimisation loop something to climb.

A published defence is public knowledge

MELON, a prompt injection defence, runs the model twice and compares tool calls to detect whether the input has redirected the agent. Its security depends on the attacker not knowing the hidden dummy prompt.

The search attack reached 76% against MELON on Llama 3.3 70B with no knowledge of the defence, finding triggers that happened to fire only in the real run. When the researchers gave the attacker the paper and the implementation, the same attack reached 95%. Same target, same method, and the nineteen-point difference is design and implementation knowledge.

This is Kerckhoffs’s principle measured on a deployed product, and it is the cleanest number I have seen for what design knowledge is worth. Assume the paper is on the attacker’s screen, because it is.

Transfer removes much of the need for target-specific access

Andy Zou and colleagues ran the largest public agent red-teaming competition to date, published as Security Challenges in AI Agent Deployment at NeurIPS 2025: twenty-two frontier AI agents, forty-four realistic deployment scenarios, 1.8 million submitted prompt injections and more than sixty thousand documented policy violations, including unauthorised data access and illicit financial actions. From the successful attacks they curated the Agent Red Teaming benchmark, which reaches near-100% attack success across nineteen models. Nearly all agents violate policy on most behaviours within ten to a hundred queries.

Two findings from that paper matter more than the headline. Attacks transfer across models and across tasks. And agent robustness shows little correlation with model size, capability, or inference-time compute budget, which means buying a bigger model is not a mitigation.

Transfer is what erodes the distinction in practice. When an attack found against one system works against yours, most of your attacker’s access to your system stops mattering, because the attack was not found there. Access still helps them confirm the transfer worked and adapt around whatever is specific to you, but your rate limits, your monitoring and your query budget are measuring the tail of the work rather than the whole of it.

Reading an access claim

Which gives a short list of questions for the next vendor deck or paper that reports a robustness number. This is the claim-evidence gap in its most specific form, because access assumptions are where vendors and researchers go wrong most reliably.

What did the attacker hold? Weights, gradients, scores, architecture, defence documentation, or a text box. If the answer is not in the paper, the number means nothing.

Access to which layer? The model, the surrounding system, or the agent with its tools and credentials. A model-level robustness result says nothing about an agent holding an OAuth token.

Was the attack adapted, or reused? A defence tested against a fixed set of published jailbreak strings has been tested against nothing. That was the whole finding: near-zero on static benchmarks, above 90% when someone designed an attack for the defence.

What was the query budget, and what did each query cost? Thirteen queries and eight hundred queries describe different threats. So does an attack that runs on the attacker’s own GPU versus one that has to come through your rate limiter.

What did it cost in utility? This one gets left out of every deck. In the AgentDojo results, putting the Protect AI detector in front of Gemini 2.5 Pro cut benign task completion from 74.2% to 39.2%, and the search attack still got through 90% of the time. Halving the usefulness of the agent while keeping the exposure is not a trade-off, and no procurement process I have seen asks for the second number.

Controls that do not depend on what the attacker knows

The constructive answer follows from the same reasoning. If you cannot bound what the attacker knows, stop building controls whose security depends on bounding it.

The twelve defences in that study were not all detectors. They fall into four groups: prompting, adversarial training, learned filtering, and schemes that depend on the attacker not knowing a secret. What they share matters more than what separates them. In every case the security property depends on a model or a classifier continuing to tell trusted instructions apart from adversarial ones, and that separation is what this entire field attacks. Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents had already made the point in Findings of NAACL 2025, bypassing all eight defences it tested at above 50%.

The alternative is architectural. CaMeL, from Debenedetti and colleagues at Google DeepMind and ETH Zürich, extracts the control flow and data flow from the trusted user query, runs untrusted data through a quarantined model with no tool access, and enforces capability-based policies in a custom interpreter at the point where a tool is called. Untrusted content cannot alter the program. On AgentDojo, in the June 2025 revision of the paper, it completes 77% of tasks with provable security against 84% for an undefended system, so the security costs about seven points of utility rather than thirty-five.

What makes that approach different is not the completion rate. Within its stated threat model, a reference monitor’s guarantee does not degrade when the attacker reads the paper, downloads the weights, or discovers a transferable trigger. The policy is enforced on the action rather than inferred from the text. That property is not unique to CaMeL. Sandboxing, capability systems, information-flow control and external authorisation all have it, and it is the oldest idea in the discipline. What it is not is unconditional: a reference monitor still falls to a policy that permits too much or an interpreter with a bug. Those are ordinary security failures with ordinary fixes, which is a considerable improvement on a control that fails whenever the attacker learns something.

Detectors still have a place. The Attacker Moves Second authors say so explicitly: they are easy to deploy and they stop unsophisticated attempts. Price them as spam filters, not as controls that close a risk.

Where this leaves a defender

Assume white-box on the model and something well short of black-box on the system. If you run open weights, the first half is already true of the base model and no amount of API hardening changes it. If you run a frontier API, the second half moves against you every time your architecture is described anywhere, including in the vendor’s own marketing.

Three things follow for Monday. Inventory your score channels and remove the ones nobody needs: log probabilities, detector confidences, verbose errors. Re-read every robustness claim in your AI security stack for its access assumptions and its query budget, and ask the vendor for the utility number alongside the attack success rate. And move the controls you are relying on from the text layer to the action layer, because a policy enforced at the tool call is the part of the stack whose strength does not fall when the attacker learns more.

The white-box and black-box labels were shorthand for a hierarchy that held in image classification and does not hold here. They are still useful for describing an experiment. They stopped being useful for deciding whether an experiment is about you.