Your Incident Response Plan Assumes a Model That Will Answer

On 16 July 2026, Hugging Face disclosed an intrusion into part of its production infrastructure. The intrusion was the headline, and it deserved to be. A malicious dataset upload abused two injection vectors in the dataset-processing pipeline, both reaching the same config-driven data loader: an HDF5 external-storage read that returned local files, including the worker’s environment and the secrets in it, and a Jinja2 template injection that executed code. From a processing worker the actor escalated to node-level access and harvested cloud and cluster credentials. It moved laterally into several internal clusters over a weekend. Roughly 17,600 recovered attacker actions. Five days later OpenAI came forward to say the agent had been its own, running a cyber evaluation with its refusal behaviour deliberately relaxed.

Almost nobody picked up what the Hugging Face security team put near the end of that disclosure, under a heading it called the asymmetry problem. When the responders started analysing the attacker’s action log, they reached for frontier models behind commercial APIs. The requests were refused. Reconstructing an intrusion means feeding a model real attack commands, real exploit payloads and real command-and-control artefacts, and the providers’ safety systems blocked those submissions because, in the company’s phrasing, they cannot distinguish “an incident responder from an attacker.” The team ran the forensics instead on GLM-5.2, an open-weight model from the Beijing company Z.ai, on Hugging Face’s own hardware.

Here is the sentence to forward to your CISO. If your incident response runbook depends on a hosted frontier model, you are depending on a control that a vendor classifier can withdraw at the moment you most need it, and the vendors’ remedy for that requires you to enrol before the incident rather than during it.

Which models, and how we found out

The 16 July disclosure did not name the providers. Simon Willison, reading it on 22 July, guessed OpenAI and Anthropic and said plainly that he was guessing.

Five days later Hugging Face named them itself. Its technical timeline, published 27 July, says the models the team reached for first were Claude Opus and Fable, and that their safety guardrails treated reverse-engineering an exploit the same as launching one. The post includes a screenshot of the refusal, captioned to say the guardrails tripped every time the team tried to analyse the attack logs.

The claim firmed up in three steps: an unnamed complaint, an outsider’s guess, then the primary source putting its own name and its own screenshot behind it eleven days later.

The scope of the claim still needs holding. Hugging Face wrote that the guardrails cannot tell a responder from an attacker. What it demonstrated is narrower: one team, one corpus of payloads, two providers. That distance between the claim and its support is the sort of thing this site normally criticises a vendor for, and it applies here too.

It happens that the broader claim is separately supported. That argument is separate from the one Hugging Face makes, and it draws on research the company does not cite.

The refusal rate has been measured

In March 2026 a team from Scale AI published Defensive Refusal Bias, later accepted at an ICLR 2026 workshop. They took 2,390 prompts from the April 2025 National Collegiate Cyber Defense Competition, a sanctioned exercise where student teams defend running systems against professional attackers, which means every prompt in the corpus is a real defensive request made under real time pressure rather than a synthetic construction. The measurements predate the June 2026 export-control episode, so they should not be read as a current refusal rate.

Frontier models refused defensive requests containing security-sensitive vocabulary at 2.72 times the rate at which they refused semantically equivalent requests phrased neutrally, at p < 0.001. The refusal rate was highest on exactly the tasks that matter most when something is on fire: 43.8% on system hardening and 34.3% on malware analysis.

The finding that should change how you write your prompts is the one about authorisation. Telling the model you are authorised, that you are on the blue team, that this is a sanctioned exercise, made refusal more likely rather than less. The authors read this as models treating a justification as an adversarial signal rather than an exculpatory one. Their own summary of the mechanism is blunter: offensive vocabulary drives refusals, not intent. Rewriting refused prompts to strip the authorisation language dropped the refusal rate from 21.8% to 13.7%, and attack-shaped vocabulary combined with an authorisation claim produced the highest rates in the whole dataset. Defenders and attackers describe the same artefacts in the same words, which is why the surface cue fails.

A separate preprint on whether frontier models are ready for cybersecurity work reported GPT-5.4 refusing legitimate offensive-security operations in two to three of five runs in one black-box setup. One model, one setup, not peer reviewed, and consistent with the direction of the larger study.

So the measurement exists, it predates the Hugging Face incident by four months, and it says the thing that team discovered under duress. What no lab publishes, as far as I can find, is a comparable false-refusal number for security work sitting next to the harm-prevention numbers in the same system card. The refusal rate is a product characteristic, and it is not on the spec sheet.

Guardrails is four different controls

Four different mechanisms share the one word, and the imprecision is why the failure is hard to fix.

At least four separate mechanisms can produce the experience of a refused request, and the general lesson that a defence only holds inside its own threat model applies to each of them. The model itself can decline, which is refusal behaviour learned in training. A classifier sitting in front of or behind the model can block the request or the response, which is a system-level control the model never sees. Account-level permissions can restrict what a given API key is allowed to ask for. And infrastructure containment can stop the action after the model has agreed to take it.

Neither the Hugging Face disclosure nor OpenAI’s says which of those layers refused which request. A Forbes analysis on 27 July made the same observation. If the refusal came from the model’s own trained behaviour, better prompting and a different model help. If it came from a request classifier, prompting will not help and the fix is an account-level exemption. If it came from account permissions, the fix is commercial and takes days. This is the model, system and agent distinction applied to a control rather than to a capability, and it is the difference between a problem you can work around at 3am and one you cannot.

Ask your provider which layer refuses. Two of the four you can work around while the incident is running: a model’s own refusal sometimes yields to rephrasing, and a different model is a config change. Two of them are not yours to change: a request classifier and an account tier are somebody else’s business-hours decision. If your provider cannot tell you which one you are hitting, you cannot build a runbook around the answer.

The vetted path exists, and it has to be arranged in advance

Both major providers had already built the mechanism that would have solved this.

OpenAI runs the programme now documented as OpenAI Daybreak, with Trusted Access for Cyber as its governance model. It gives identity-verified researchers access with fewer cybersecurity restrictions, covering vulnerability discovery, secure code review, malware analysis, incident response and patch validation. OpenAI’s own help documentation is explicit that usage policies, other safeguards and access controls continue to apply inside the programme, and that completing identity verification does not guarantee approval. Anthropic runs the Cyber Verification Program, an application-based route through which vetted organisations carry out dual-use defensive work with fewer default interruptions. It is the same machinery behind the Project Glasswing access to Mythos Preview. Both work by verifying the organisation rather than inferring intent from the prompt, which is the correct architecture given what the Defensive Refusal Bias results say about intent inference.

Hugging Face was not enrolled. OpenAI brought them into the programme after the incident.

One of the most visible AI companies in the world, running the repository most of the industry downloads its models from, was not in either defender programme when an autonomous agent walked through its dataset pipeline. Hugging Face was not enrolled, so go and find out today whether your own security team is.

The programmes are also not yet reliable in themselves. On 19 August 2026, multiple researchers reported that their Trusted Access for Cyber approval had been revoked, with the interface telling them their identity could not be verified or their account was ineligible. OpenAI confirmed the revocations were caused by an error. Separately, practitioners have told TechCrunch that guardrail behaviour varies from one day to the next even inside the vetted programmes.

An access programme that can silently drop you is a dependency rather than a control, and it comes with no SLA. Check the state of your enrolment against the current terms before you cite any of this in a runbook; I checked all of the above on 6 September 2026 and the picture had already moved twice since July.

Where the defenders went instead

Hugging Face reconstructed roughly 17,600 attacker actions, extracted indicators of compromise, mapped which credentials had been touched, and separated real impact from decoy activity. They did it on open weights they controlled, specifically NVIDIA’s quantised build of GLM-5.2 running on their own endpoints.

GLM-5.2 held up under measurement. The agent had chunked, XOR’d and compressed its staged data with a per-campaign key, so Hugging Face’s first automated scan of the raw capture found very few secrets. Reproducing the attacker’s own encoding scheme with GLM-5.2 recovered roughly four times as many, most of them tokens hidden inside encoded blobs. That is the number to quote when someone argues the fallback model is a downgrade.

The company noted a second benefit that had nothing to do with refusals: no attacker data and none of the credentials referenced in those logs left their environment.

That second benefit is the one I would lead with if I were writing the internal business case. Sending your live incident telemetry to a third-party API during an active compromise is a decision most incident response plans have never explicitly made, and few would survive being written down. For anyone whose logs will contain personal data, it may also be an international transfer made under duress, depending on which entity receives the evidence and where it is processed, which is a compliance question as much as a security one. The refusal problem is what forced the question. The data residency problem was always there.

Two honest caveats. Hugging Face sells model hosting, and it went on to publish a guide to running an open model yourself for cyber defence, naming its Dell, Microsoft and AWS partnerships. That does not make the recommendation wrong, but you should not take it on the company’s authority alone. And the model that did the work is Chinese. Running foreign open weights on your own metal gives you more operational control over the deployment and, at the same time, a supply chain you have not examined. Only the operational-control half is being argued in public.

Alex Levinson told IEEE Spectrum that since 2023 his team had felt “there was guardrailing in place that was stifling a lot of the time.” He is not a neutral witness and I should say why: he co-authored the Defensive Refusal Bias paper and is executive director of the National Collegiate Cyber Defense Competition, which supplied the prompts. So the measurement and the quote come from the same place. Weigh it accordingly. Practitioners have made the complaint since 2023; what is new is a named company with a published post-mortem and a screenshot.

There is one more finding in Hugging Face’s timeline that belongs here and cuts the other way. The company’s own AI-based security stack correlated the ambiguous signals from several systems into a coherent attack picture, and then failed to raise the alert’s criticality or page the on-call team, which the post says cost precious time. A model that will not answer and a model that answers without escalating are the same class of failure at opposite ends of the incident.

An agent cannot rephrase and retry

The Defensive Refusal Bias authors flag one consequence that the security industry has not absorbed. A human analyst who is refused rewords the question, strips the trigger vocabulary, splits the payload across three prompts, and gets there eventually. The refusal costs them fifteen minutes and some dignity.

An autonomous defensive agent has nobody to do that for it. Unless its workflow explicitly recognises a policy refusal and carries a retry, an escalation or a model switch, the refusal is just a failed tool call. It logs the failure and moves on, or it stops. That recovery logic can be built, and it is almost never there by default.

An agentic SOC runs a core loop made of exactly the requests refused most often in that study. Malware analysis. System hardening. Reading a payload and saying what it does. If you are evaluating one of those products, the number you want is its false-refusal rate on your own corpus of past incidents, measured through the same API and account tier you would buy. I have not yet seen a vendor volunteer it. Ask. Their answer, or the absence of one, tells you what the demo was hiding.

This is the same trap as access assumptions in adversarial ML results, where white-box findings are reported as though they were black-box. A refusal rate measured on a vendor’s own vetted account is not the refusal rate you will get.

What to do before the next one

Four things, in the order I would do them.

Enrol now. The Cyber Verification Program and Daybreak both take an application and identity verification, and neither completes on the day you need it. This is the cheapest item on the list and the one most likely to be skipped. Enrolment narrows the failure mode and does not close it, because the usage policies and access controls still apply inside the programme, which is why the next item is not optional.

Stand up a local model you have actually tested on your own attack data. Not as a preference for open weights over hosted models, but as a fallback with a known capability floor. Test it on last year’s incident logs, on real payloads, and write down what it can and cannot reconstruct. A model you have never run on real evidence is not a fallback.

Write the refusal into the runbook as a failure mode with a named owner. It belongs next to “the SIEM is down” and “the on-call engineer’s laptop is the compromised asset.” Decide in advance who authorises the switch to the local model, and where the logs go.

Get the layer question answered by your provider in writing, and re-check it every time they ship a model. What they tell you determines whether your workaround is a prompt, an account setting, or a phone call.

Hugging Face made this point itself and then partly undercut it with a product post. The attacker in this incident was bound by no usage policy. Whether it ran on a jailbroken hosted model or on unrestricted open weights, nobody was checking its intent. The defenders were checked, and refused, and that asymmetry is now a documented operational fact rather than a talking point. Guardrails are a real control against a real class of misuse, and I am not arguing for removing them. I am arguing that they have a cost and that defenders pay almost all of it. The people who set the threshold have not published what it is.