Your Model Hub Scans for Malware, Not for Behaviour

The move to open weights is not coming, it has happened. Mistral now hosts GLM-5.2, a Chinese open-weight model, alongside its own. When Hugging Face’s incident responders set out to reconstruct an intrusion of their own production infrastructure, the commercial models they tried refused to analyse the attack. They finished the job on GLM-5.2, running on hardware they controlled.

Both of those decisions are defensible on their own terms. Holding and running the weights removes the provider’s hosted API as a point of withdrawal, repricing or policy change. Anyone who watched two frontier models get withdrawn following a government directive, reportedly against a ninety-minute compliance deadline, understands the appeal. What it does not remove is refusal behaviour already learned into the checkpoint, which travels with the weights.

What comes with those weights is not what the debate is about. Researchers have implanted conditional backdoors into open-weight models using ordinary fine-tuning and parameter-efficient methods, with no pretraining run required. Some constructions survive the retraining you would do to remove them and stay unobtrusive on any evaluation that does not present the trigger. No control that a model hub currently runs will tell you whether a given checkpoint does that.

Not one of those facts depends on where the model was trained.

The claim that gets made, and what supports it

The objection to Chinese open weights, as it is usually put, is that a state actor could have put something in the model. It is offered as a reason not to adopt, and it almost never comes with a demonstrated instance.

Hold it to the standard this site applies to vendor claims. What was demonstrated, on which model, under what access assumptions, verified by whom? I could not find a publicly documented, independently verified case in which a released Chinese open-weight base model was shown to contain a state-implanted conditional backdoor. That is not proof that none exists, and I am not offering it as one. It means the specific allegation has not met the evidentiary standard we would demand of any other technical security claim. That limits what can be asserted about implanted backdoors. It does not settle the separate questions of supplier governance, jurisdiction or strategic dependence, which are legitimate and answered by different evidence.

Two things follow, and they point in opposite directions from the usual conclusion.

The first is that provenance is being asked to do work it cannot do. Provenance is verifiable within limits: signed commits, publisher identity, hashes. What it establishes is who published an artefact. Integrity establishes that you received the same artefact they published. No check at either layer establishes that the policy learned inside the weights is benign. If your assurance argument is “we use a model from an allied country,” you have answered a supplier-governance question and left the behavioural one open.

The second is more uncomfortable. The attack everybody is worried about in the abstract has been built, published and measured by academic researchers, across multiple model families, and it works.

What has actually been demonstrated

A March 2026 preprint implanted temporal backdoors into a tool-using model through parameter-efficient fine-tuning. The method separates the implantation of the hidden capability from the alignment that makes the model behave normally at all other times, which is why the resulting model still works. It is not untouched: the paper reports TruthfulQA falling from 0.81 to 0.57.

Tested on Qwen3-4B-Thinking against a private set of 100 synthetically generated terminal tool-calling tasks, the backdoor fired at a 99.6% mean attack success rate when the system prompt supplied a 2026 date, against a 0.7% false-positive rate on dates from 2015 to 2025. The trigger is a date the prompt hands the model, and no machine clock is involved. The variant refined with reinforcement learning reached 100% at several settings in a temperature sweep running from 0.0 to 2.0, not across the whole range, and that refinement belongs to building the attack; it is not evidence that the backdoor survived a defender’s remediation.

Two limits belong in the same breath as those numbers. The evaluation set is private, so nobody outside the group can reproduce that exact benchmark, though the method itself is described well enough to reproduce. And these are the authors’ own measurements of their own construction.

The fact that should change procurement thinking is that implanting this requires no pretraining run. Parameter-efficient fine-tuning, the same tooling used to produce every community derivative and adapter on every model hub, is sufficient. One practitioner reproducing the poisoning setup Microsoft used to evaluate its scanner reported training a LoRA adapter on Mistral-7B that held 0.58% as many parameters as the base model, at the cost of one line of data preprocessing over a fine-tune they were running anyway. That figure counts trainable adapter parameters, not how many weight entries the attack alters once the adapter is merged. A low-rank update can move a great many of them. The effort figure describes one experiment and does not include the cost of developing, testing and distributing an attack, and it comes from a personal write-up rather than a reviewed paper.

The capability is neither exotic nor expensive, and it does not require a state. It requires someone who publishes a fine-tuned derivative or a LoRA adapter that people download because it performs well on a leaderboard.

The base models get scrutiny. The derivatives and adapters organisations actually deploy get a benchmark score and a download count. This is inherited risk in its purest form, and it is the same fragmentation problem this site has been describing since 2023. None of it cares what flag the base model flies under.

Fine-tuning is not proof of backdoor removal

The instinct on hearing about a backdoored model is to fine-tune it on your own clean data and move on. The research does not support the instinct, and it stops short of the opposite blanket claim.

Anthropic’s 2024 sleeper-agents work trained models to write secure code when the prompt indicated one year and exploitable code when it indicated another, then applied the standard remediations: supervised safety fine-tuning, reinforcement learning from human feedback, and adversarial training. The behaviour persisted through all three, under the procedures they tested. Two of its secondary findings matter more than the headline. Robustness to safety training increased with model scale across the sizes they tried, which is a trend in that study rather than a law about capability. And adversarial training did not remove the backdoor; it taught the model to recognise the adversarial probe, which made the behaviour harder to detect and left it in place.

Later work extends this to the case you would actually face, where the attacker implants the backdoor before release and has no control over the fine-tuning the downstream user performs afterwards. Be precise about what it shows. Ordinary implants often degrade under that treatment, sometimes to nothing. Implants designed to survive it can hold at very high rates. Persistence is not automatic; it is available to an attacker who wants it.

So a further round of clean training is not, by itself, evidence that a model has come out backdoor-free. Some implants weaken or disappear under it; others are built to survive it. And in the adversarial-training case Anthropic tested, the training made the backdoor harder to find while leaving it in place. A removal claim needs an evaluation against the relevant attack, not a record that another training run happened.

Your supply-chain controls are checking the wrong layer

Model hubs are not doing nothing. Hugging Face runs malware scanning, pickle and import analysis, secret scanning and third-party checks from Protect AI and JFrog, and it has pushed the industry hard toward safer serialisation formats, which closed a real and previously exploited class of attack where loading a model executed code. Safer formats narrow the code-execution routes without closing every one.

Those controls are about the artefact. Everything above is about the behaviour the artefact encodes.

There is currently no equivalent of a signature check for the proposition that this set of weights behaves differently when a particular string, date or context appears. A model that passes every file-format check, loads cleanly, scores well on your benchmark suite and answers your evaluation set correctly can still produce the behaviour described above, because none of those tests presents the trigger.

Say the gap plainly. Model hub checks can find malicious files, unsafe serialisation and some kinds of embedded code. They do not establish that the learned behaviour of an otherwise valid checkpoint is free of conditional backdoors. Artefact security and behavioural assurance are different questions, and conventional vulnerability management still applies to the loaders, libraries and other software around the model. A checkpoint can be authentic, intact and free of embedded malware while the policy learned inside it is not trustworthy, and no existing check reaches that.

The archive on this site has drawn the distinction between data poisoning and backdoors for years. It needs stating carefully, because backdoors are usually installed through poisoning; the two are not alternatives. The useful contrast is between broad or untargeted poisoning, which shifts behaviour across a distribution, and a trigger-based backdoor engineered to leave the model normal outside a narrow condition. Standard evaluation is far more likely to expose the first, because the second is built to survive tests that never present the trigger.

Detection exists. Certification does not.

The situation is better than the sections above imply, and that changes where you should spend.

The most substantial work here is The Trigger in the Haystack, published by Microsoft’s AI red team in February 2026. It is a scanner and not a monitor, and two properties make it interesting. It assumes no prior knowledge of the trigger or the target behaviour, which is the realistic condition, since a defender who already knows the trigger has solved the problem. And it runs on inference operations alone, with no retraining. That is not a black-box test: the setup assumes access to the model files, weights and tokenizer included, and it reads output distributions and attention-derived signals. Treat it as a research scanner for specified sleeper-agent-style backdoors, run with model-internal access, rather than a deployable cleanliness check.

The method uses two properties of backdoored models. Backdoored models often memorise their poisoning data, which means the examples can sometimes be pulled back out using memory-extraction techniques. And a poisoned model’s output distributions and attention heads behave distinctively when a trigger is present in the input. From those, the paper reports recovering working triggers across several backdoor scenarios and a range of models and fine-tuning methods.

A much smaller study circulates with more precise-looking numbers attached, and industry research notes are citing it as though it settled something. It is a University of Windsor course project, evaluated on one deliberately backdoored model across forty responses, and its reported recall and F1 do not reconcile with the confusion matrix it prints. I am not repeating its percentages here, because a number that cannot be derived from the counts beside it should not be in circulation at all. It is a reasonable student exercise and it is not a basis for a procurement decision.

Scanning and runtime enforcement do different jobs, and I nearly treated them as one control. Scanning is an admission control: you run it before a model enters your estate, it looks for a latent property, and it can miss. Runtime enforcement is different: it assumes the scan missed and constrains what the model is allowed to cause. You need both, because behavioural scanning is now good enough to belong in model admission and nowhere near good enough to certify a model as clean.

What to do

Four things, and the ordering is deliberate.

Treat model weights as an untrusted component, in the same sentence as a third-party binary. This is a classification decision, not a technical one, and everything else follows from it. It applies identically to Qwen, Llama, Mistral and to the fine-tuned derivative of any of them that your data science team found last month.

Put the control at the tool call, not at the model. The demonstrated payload in the agentic case is not bad text, it is a tool invocation the model would not otherwise make. Enforce permissions and action constraints outside the model, before the call executes, and verify them independently of what the model says it is doing. Those controls can stop specified harmful actions even when the backdoor is never detected, though they do not certify the model and their reach ends where the policy stops. An authorised write can still carry the wrong value, and an approved destination can receive data that should never have gone there.

Inventory your derivatives and adapters separately from your base models. Most organisations can name the base model. Far fewer can produce the list of LoRA adapters and community fine-tunes running in production, who published them, and what evidence exists for any of it. That list is your actual exposure, and most organisations have yet to assemble it.

Do not let national origin substitute for technical assurance. Jurisdiction, export exposure and supplier governance are legitimate procurement questions, and I am not arguing that anyone should stop asking them. They are not behavioural assurance, and treating them as such gives you a confident answer to a question no evidence currently addresses. A backdoored Llama derivative published by an anonymous account and a clean Chinese base model are both entirely plausible. A behavioural control should be able to tell a malicious model from a benign one, with national origin nowhere in the verdict.

The uncomfortable part of this is that the buyers moving to open weights are mostly right about why. Withdrawal risk is real, the refusal problem is real, and running a model you control removes the first outright and only sometimes helps with the second. What they have done is trade a dependency they could see for one they cannot. The industry has spent its attention arguing about the passport on the model card, in both directions, instead of building the thing that would actually check.