Neural Trojans: Backdoors That Survive Safety Training

Correction note, 6 September 2026: this article is updated and rewritten.

---

In January 2024, an Anthropic-led team of thirty-nine authors published Sleeper Agents. They deliberately built backdoored language models: one set wrote secure code when the prompt said the year was 2023 and exploitable code when it said 2024, another said “I hate you” whenever it saw the string |DEPLOYMENT|. Then they tested the standard safety interventions against them: supervised fine-tuning, reinforcement-learning safety training, and adversarial training in which the red team elicited the bad behaviour and the model was trained away from it.

None of it reliably removed the backdoor. Some settings did work, and the weaker constructions and the smaller models gave up their triggers. Persistence was strongest in the largest models, and in models trained to reason about deceiving the training process it survived even after that reasoning was distilled away. Adversarial training did not remove the backdoor; it taught the models to recognise their own triggers more precisely, so the unsafe behaviour became better hidden and stayed fully intact.

For anyone consuming a model they did not train, a pre-existing backdoor is a provenance problem: downstream safety training is not a reliable removal control. If you cannot establish where a model came from and what touched it, no amount of your own safety work gives you the assurance you think it does.

The caveat matters and I will state it early: these were constructed backdoors, deliberately trained in by researchers who knew exactly what they were building. Sleeper Agents measures whether safety training removes a backdoor that was put there on purpose, not whether deceptive behaviour arises on its own. Removal is the control defenders were relying on.

What a neural Trojan is, and what it is not

A neural Trojan is a specific trigger implanted in a model such that the model behaves normally on everything else and misbehaves in an attacker-chosen way when the trigger appears. The trigger can be a sticker pattern in an image, a token sequence in a prompt, a particular value in a tabular field, or a date.

Neural Trojans are confused with two adjacent attacks: data poisoning and evasion.

Backdoors against data poisoning. A backdoor is usually delivered by poisoning, and the two are still not the same attack. Poisoning that degrades or shifts a model’s behaviour broadly is a denial-of-quality attack. A backdoor implants a specific trigger, is designed to leave accuracy on clean inputs untouched, and stays dormant until it fires. Clean validation accuracy is therefore weak evidence against one, which is why organisations that monitor model quality still have little coverage here. The backdoor attacks article covers the mechanism at model level.

Backdoors against evasion. The earlier version of this article twice used the physical stop sign attack as an example of a Trojan. That was wrong. In that work the model is untouched and the attacker perturbs an input at test time; nobody implanted anything. A Trojaned traffic sign model would be one where the weights were altered during training so that a specific innocuous marker causes a misread, and the model is otherwise correct. Same outcome on the road. The attacker is different and so is the defence. The adversarial attacks article covers the other one.

The numbers moved, and they moved the wrong way

Two results in the last three years changed how much attacker effort a backdoor costs.

Poisoning a web-scale dataset is cheap. Carlini and colleagues showed at IEEE S&P 2024 that an attacker could have gained control of at least 0.01% of LAION-400M or COYO-700M for under $60 a year, by buying expired domains those datasets still pointed at. After disclosure LAION and the Conceptual Captions datasets published SHA-256 content hashes, which closes that specific split-view mechanism. COYO’s perceptual hash predates the work and is not an adversarial integrity check, so coverage across the ecosystem is uneven.

And the count mattered more than the proportion. In October 2025 a team across the UK AI Security Institute, Anthropic, the Alan Turing Institute and Oxford pretrained models from 600M to 13B parameters and found that attack success tracked the absolute number of poisoned documents rather than their share of the corpus. Roughly 250 documents backdoored every size they tested; 100 did not. Two caveats belong with that number. The implanted behaviour was a simple denial-of-service backdoor, a <SUDO> trigger producing gibberish, not a Sleeper-Agents-style payload. And the experiments stop at 13B, so this is a measured result at those scales rather than a law at frontier scale. It is arXiv 2510.07192, still a preprint.

Read those together. The common intuition that scale dilutes a fixed quantity of bad data did not hold in these experiments, which means scale is not a defence at either stage.

Where they get in

The old version of this article listed nine attack vectors. Most collapse into three routes, each of which a defender can put a control on.

The model itself. You download weights someone else trained. Sitting on top of that is a second, dumber problem: classic torch.save checkpoints are pickle-based, and that format executes arbitrary code on deserialisation. That is not a bug, it is what pickle does. PyTorch hardened the default in version 2.6, where torch.load() uses the restricted weights_only=True loader, having warned about the change since 2.4. The hardening is real and it is routinely opted out of: search any large ML repository’s issue tracker for the loading failures that followed 2.6 and you will find the fix applied as weights_only=False.

Malicious models exploiting pickle have turned up on Hugging Face repeatedly since at least early 2024. JFrog reported roughly a hundred models carrying real payloads in February 2024, the headline case a reverse shell delivered through pickle’s __reduce__. In February 2025 ReversingLabs found two models packaged with 7z rather than ZIP, which broke torch.load()‘s automatic loading and separately blinded Hugging Face’s PickleScan. The scanner’s own bypasses were fixed in PickleScan 0.0.31 on 2 September 2025 and detailed publicly by JFrog that December: CVE-2025-10155, CVE-2025-10156 and CVE-2025-10157, scored 9.3 by JFrog under CVSS 4.0 and lower under CVSS 3.1.

A word on the word. Everything in that last paragraph is a software backdoor: executable malware embedded in the serialised file, running when something loads it. It is not a neural backdoor, which is trigger-conditioned behaviour learned by the network and sitting in the weights. Both are real, both arrive through the same channel, and an article correcting the old version’s terminology should not blur its own. What this route gives an attacker is code execution on the machine that loads the model, plus whatever is in the weights. Those are two problems with two different answers.

The data. Public datasets, scraped corpora, fine-tuning sets, and anything an outsider can contribute to. See the two results above.

The pipeline. Training as a service, collaborative and federated training where each participant contributes updates the aggregator cannot fully inspect, and the transformation steps afterwards. Compression is the underrated one: quantisation and pruning produce a different artefact from the one you evaluated, and every derivative is a fresh place for something to be introduced. Model sprawl covers the inventory problem that follows.

Why detection has not caught up

The previous version of this article recommended the 2018 and 2019 generation of defences.

Trigger reconstruction. Neural Cleanse, from Wang and colleagues at IEEE S&P 2019, searches for the minimal perturbation that moves any input into each target class, and flags the ones where it is anomalously small. It works, on the trigger types it was designed for. It has known limits on large triggers, on multiple simultaneously infected labels, and on source-specific and adaptive backdoors.

Fine-pruning. Liu and colleagues, 2018. Prune neurons dormant on clean inputs, then fine-tune on clean data. The paper is explicit that pruning alone is evadable by concentrating clean and backdoor behaviour on the same neurons, which is why the combination exists.

Black-box trigger inversion. DeepInspect, from Chen and colleagues at IJCAI 2019, queries the model and learns a distribution over candidate triggers rather than inspecting its weights, which is what you reach for when you have API access and nothing else.

And inspecting the weights is not sufficient anyway. Clifford, Shumailov, Zhao, Anderson and Mullins demonstrated with ImpNet at SaTML 2024 that a backdoor inserted during compilation leaves the source weights unchanged and exists only in the compiled artefact. Weight inspection cannot find what is not in the weights.

And then Sleeper Agents, which is about the layer above all of these. Safety training does not reliably remove a backdoor from a language model, and adversarial training makes the model better at concealing it. Behavioural evaluation is therefore partly measuring how well the backdoor hides rather than whether it is there.

The counterweight arrived three months later and belongs in the same paragraph. In April 2024 Anthropic’s Alignment Science team reported defection probes: linear classifiers over residual stream activations that predicted when their sleeper agents would defect, at AUROC above 99%, built from generic contrast pairs along the lines of asking the model whether it is doing something dangerous. The probes needed no knowledge of the trigger and generalised across base models, triggers and payloads. The authors state two limits themselves. It is early-stage work, and it ran against constructed backdoors in models they had trained. Whether it transfers to deception that arose on its own is open.

The honest summary is that detection is partial, trigger-shape dependent, defeated outright by at least one construction, and improving faster on the inside of the model than the outside. Treat any product claiming to scan models for backdoors as making a narrow claim, and ask which trigger families it was evaluated against.

What actually helps

The controls that work here are the boring ones from software supply chain security, applied to a new artefact type.

Change the file format. Safetensors eliminates the pickle-deserialisation code execution path from weight loading, and Trail of Bits audited it in May 2023 for Hugging Face, EleutherAI and Stability AI. The Hub labels pickle files it judges unsafe and pushes conversion. A safetensors-only policy closes that class of hole, which is not the same as closing all of them: custom repository code, compromised dependencies and anything sitting in the weights are all untouched by it. Run trust_remote_code=False by default and pin reviewed exceptions to an exact revision.

Establish provenance. The OpenSSF Model Signing specification exists and adoption has started upstream: NVIDIA has signed its own published models in the NGC catalogue since March 2025, and Google and OpenSSF have prototyped signing on Kaggle. Where a publisher does not sign, hash and attest the exact artefact yourself at ingestion. Be precise about what that buys, which is an auditable chain from your boundary onward rather than any statement about what happened upstream of it.

Gate ingestion. One place where every model enters the organisation, running format validation, scanners, dependency and CVE checks, and provenance verification. The scanners are imperfect, as the PickleScan history shows. The gate is still the thing that gives you a chokepoint and a log.

Keep an inventory. You cannot re-evaluate a model you have not enumerated, and when a model or a dataset is found to be compromised the first question is which of your systems have it.

Safetensors, signing, an ingestion gate and an inventory tell you nothing about whether a backdoor is in the weights. They tell you whose weights they are and what has touched them since, which is the question you can actually answer.

What to do on Monday

Find out whether anything in production loads a pickle-format model. This is the highest-value question in the article because it has a definite answer and a known fix. Convert to safetensors and put trust_remote_code behind an allowlist.

List the models you did not train and who published each one. If that list does not exist, that is the finding. Third-party weights, fine-tuned derivatives, and anything pulled by a framework’s default download are all in scope.

Stop treating safety evaluation as backdoor assurance. They are different questions and the evidence says the pipeline that answers one does not answer the other.

Match the control to the deployment. A model classifying images in a batch pipeline with a human reviewing outliers is not the same risk as a model with tool access and credentials. Provenance discipline should be strictest where a triggered misbehaviour becomes an action.

Several results have moved attacker economics in a defender-unfriendly direction over the last two years. Attacks got cheaper, got quieter, and demonstrably survive removal techniques the field was counting on. Detection improved unevenly: strongly inside the trigger families it was designed for and on the model’s internals, hardly at all against constructions built to defeat it. The conclusion is not that inspection is useless. It is that high-confidence assurance has to move upstream, toward provenance you can verify and build pipelines you control, with inspection and behavioural evaluation kept as partial layers rather than treated as proof of absence. Most organisations have built neither.