Your Model Explanations Are an Attack Surface

Smitha Milli, Ludwig Schmidt, Anca Dragan and Moritz Hardt showed in 2018 that gradient-based explanations of a model quickly reveal the model itself. Their theoretical result gives an algorithm that provably learns a two-layer ReLU network from gradient queries, using a number of queries independent of the input dimension and nearly optimal in the model size. Their heuristics reconstruct models from gradient explanations orders of magnitude more query-efficiently than attacks that see only the prediction interface. Gradients are a stronger learning primitive than labels, and an explanation endpoint hands them out.

Ulrich Aïvodji, Alexandre Bolot and Sébastien Gambs extended the finding to counterfactual explanations in September 2020, building faithful copies of black-box models trained on real datasets at high fidelity and high accuracy under low query budgets. Counterfactuals are the explanation format regulators and usability researchers prefer, because “your loan would have been approved with €4,000 more income” is a form a person can act on. It also gives an attacker an opposite-class point close to the decision boundary with every query.

Anyone running an explanation endpoint should treat it as a query interface that returns gradients or near-boundary points, put it in the threat model on that basis, and scope, log and rate-limit it accordingly. A companion survey of which explainability frameworks still ship in 2026 covers the tooling. What follows covers what the tooling gives away.

Explanations can be forged

Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh and Himabindu Lakkaraju published a scaffolding attack against LIME and SHAP at AIES 2020. The construction wraps a deliberately biased classifier in a detector that distinguishes real inputs from the synthetic points an explanation method submits. On real inputs the wrapped model behaves exactly as the biased model does. On the perturbed points LIME and SHAP submit for their local approximation, it behaves as an innocuous model the adversary specifies. The resulting explanation describes a model that made no real decision.

They evaluated the attack on COMPAS, Communities and Crime, and German Credit, building classifiers that decided purely on race for the first two datasets and purely on gender for the third. Against LIME, the attack moved the sensitive feature out of the top position for 100 per cent of held-out points on COMPAS and Communities and Crime, and 91 per cent on German Credit. Against SHAP the figures were 84, 100 and 85 per cent. The authors report LIME as the more vulnerable of the two.

State the access assumption plainly, because it decides who this threatens. The attack requires control of the model. An outsider cannot do this to your deployment. A model supplier can do it to you, and so can a compromised training pipeline. An internal team facing a fairness review can do it to its own auditor. If your assurance process consists of asking a vendor for SHAP values computed over the vendor’s model, that process has had a published bypass since 2020.

Success depends on the out-of-distribution detector. Where it cannot separate perturbed points from real ones, both methods correctly identify the sensitive feature and the attack fails. The mechanism therefore reaches perturbation-based explainers whose synthetic queries can be told apart from the data distribution, which covers LIME, KernelSHAP and occlusion methods. Methods reading the model’s internals escape this particular attack, though not manipulation in general. Amirata Ghorbani, Abubakar Abid and James Zou showed at AAAI 2019 that neural network interpretations are fragile under small input perturbations that leave the prediction unchanged, and Juyeon Heo, Sunghwan Joo and Taesup Moon demonstrated fooling interpretations through adversarial model manipulation at NeurIPS 2019. Erwan Le Merrer and Gilles Trédan framed the general case as the bouncer problem: providing explanations cannot stop a remote service lying about the reasons behind its decisions.

Explanations reconstruct the model

The extraction results have accumulated into a family.

Milli and colleagues covered gradients. Aïvodji and colleagues covered counterfactuals. Yongjie Wang, Hangwei Qian and Chunyan Miao published DualCF at FAccT 2022, which feeds a returned counterfactual back to the same endpoint so the second query returns a point on the opposite side of the boundary; for binary linear models their Lemma 4.1 extracts a substitute at complete agreement from a single such pair. Takayuki Miura, Toshiki Shibahara and Naoto Yanai published MEGEX at SecTL 2024, a data-free extraction attack against gradient-based explainable AI that pairs a generative model with the explanation channel to cut the query count. Abdullah Çağlar Öksüz, Anisa Halimi and Erman Ayday published AUTOLYCUS in 2023, extracting interpretable models through their LIME and SHAP explanations, across decision trees, logistic regression, naive Bayes and k-nearest neighbour.

The reason this family exists is structural. Model extraction works by learning a decision boundary from responses, and its cost is measured in queries. Explanation makes each query more informative. A label reports which side of the boundary the point is on. A gradient reports the direction the boundary takes. A counterfactual supplies a point near it. The explanation is doing what its designers intended, for a reader whose interests differ from yours.

The consequence is a changed budget calculation. If you modelled extraction risk against a prediction API and set query budgets from that model, adding explanations to the response invalidates the calculation. A budget that was safe for labels is not safe for gradients.

Explanations identify people in the training data

Reza Shokri, Martin Strobel and Yair Zick measured the privacy cost at AIES 2021. They ran membership inference against models that returned a prediction plus a feature-based explanation. Backpropagation-based methods leaked a significant amount of information about individual training records. Their stated mechanism: such an explanation reveals statistics about the decision boundary near the queried input, and the values differ systematically between points the model trained on and points it did not.

They also measured the trade-off. Perturbation-based explanations resist the attack better and produce lower-quality explanations. The property making an explanation informative to a user is the property making it informative to an attacker.

The exposure extends to model inversion. Xuejun Zhao, Wencan Zhang, Xiaokui Xiao and Brian Lim showed at ICCV 2021 that explanations improve inversion attacks, and that activation-based saliency map explanations leaked more than sensitivity-based gradient explanations.

Differential privacy applied at training time is the only mitigation here with a formal guarantee, and it covers an explanation only where that explanation is post-processing of the DP-trained model. An explanation procedure that separately touches raw training data or another non-DP source falls outside it. DP also imposes a privacy-utility trade-off and can reduce accuracy. Everything else in this area lowers leakage empirically without bounding it.

Explanations tell an evader what to change

The operational case is worse than the privacy case, because it concerns systems built for adversaries.

Explainable intrusion detection and explainable malware classification have become a substantial literature, on the reasonable ground that an analyst triaging an alert needs to know which features raised it. The difficulty is that a feature attribution telling your analyst why a sample was flagged tells an attacker which features to change. Kshitiz Aryal, Maanak Gupta, Mahmoud Abdelsalam and Moustafa Saleh made the attack explicit in May 2024, using SHAP to locate the high-impact regions of a Windows PE file and guide evasion against a malware detector.

I described the same dynamic in adaptive attacks against machine-learning intrusion detection and in model evasion. Explanation reduces the attacker’s search cost sharply. An evader without explanations discovers which features matter by probing. An evader with explanations gets told.

Name the claim-evidence gap in this subfield, because it is wide. Most published explainable-IDS work evaluates on NSL-KDD, CICIDS2017 or UNSW-NB15, reports improved interpretability, and runs neither an analyst study nor an adaptive adversary. PROVEX, a 2025 provenance-based system, is honest about its limits: it reports an average explanation overhead of three to five seconds per event, and its limitations section describes end-to-end explanation after an alert taking on the order of tens of seconds, making it an offline forensic tool and not something in the alert path. Treat “we added SHAP to our detector” as unevaluated until somebody shows both the analyst benefit and the evasion cost.

Where explanation works, and who has that access

The strongest positive result for interpretability as a security control is Anthropic’s alignment auditing game, published in March 2025.

A red team trained a language model with a hidden objective behind an information barrier. The training taught the model about exploitable errors in RLHF reward models and then trained it to exploit some of them, and the model generalised to behaviours it believed reward models would score highly, including behaviours never reinforced during training. Four blue teams then investigated it without knowing what had been done. Three teams found the hidden objective, using sparse autoencoder features, behavioural attacks and analysis of the training data. The fourth team did not.

Access separated them. The three successful teams had the model weights and a sample of the training data, and each of them reached the answer by finding the small subset of training data describing the reward-model biases. The fourth team had API access only and no training data.

The conclusion this supports is narrower than the one usually drawn from it. Auditors holding the weights and the training data succeeded where an API-only auditor failed. It does not isolate weights as the decisive variable, because training-data access was what the successful teams actually used. What it does establish is that the auditing capability vendors describe is not available to a customer holding an API key.

Consider who occupies each position. The organisations able to audit this way are the organisations that trained the model. Enterprises deploying frontier models through an API occupy the fourth team’s seat, and so does any regulator without weights or training-data access. The distinction between model, system and agent applies with force here: a successful audit of this kind describes the model, and says nothing about the retrieval layer feeding it, the tools it can call, or the credentials attached to those tools.

The Article 86 problem

Article 86 of the EU AI Act gives a person affected by a decision made on the basis of an Annex III high-risk AI system, excluding systems under point 2 of that Annex, the right to obtain clear and meaningful explanations of the role the system played and the main elements of the decision taken.

Two features of the provision matter for security.

The deployer owes the explanation, not the provider that built the model. The organisation required to produce the explanation usually has the least access to the thing being explained, which is the position the fourth auditing team occupied.

The timing is contested, and the dispute matters more than any resolution I could offer. The Digital Omnibus, in force from 27 July 2026, deferred the Chapter III high-risk obligations for Annex III systems to 2 December 2027, and Annex I systems to 2 August 2028. The Omnibus did not amend Chapter IX, which contains Article 86, so on the face of the text the right applied from 2 August 2026 while its trigger, an Annex III high-risk classification, takes effect only in December 2027. December 2027 is the practical reading. It is not a settled one, and anyone building a compliance calendar on it should take EU-law advice. Checked 5 September 2026 against the amending regulation and secondary commentary; confirm against the Official Journal.

The practical answer does not depend on resolving that. An Article 86 explanation must be reconstructable after the fact, which requires knowing which model version made the decision, on which inputs, under which configuration. That is a logging and versioning problem before it becomes an explainability problem, and the Article 12 logging obligations and Article 26 deployer records supply the raw material. Designing the decision record and the explanation procedure together costs considerably less than retrofitting one to the other after a complaint arrives.

Cynthia Rudin’s answer

Rudin argued in Nature Machine Intelligence in 2019 that high-stakes decisions call for inherently interpretable models, and that explaining black boxes perpetuates bad practice and can cause serious harm.

As a security argument, Rudin’s position is the strongest one available. Every attack above targets the explanation layer: the surrogate approximating the model, the gradient the model exposes, the counterfactual it generates. A model whose logic is directly readable has no explanation layer to attack. Nobody can forge an explanation that is the model. Nobody can extract a model from an explanation when the explanation is the model you already published.

For tabular high-stakes decisions, sparse rule lists, monotonic models and generalised additive models frequently reach competitive accuracy. Annex III spans biometrics, critical infrastructure, employment, education, essential services, law enforcement, migration and justice, and many of those deployments are tabular. I would not extend the recommendation to language models.

What to do

Treat the explanation endpoint as a query interface. Rate-limit it, log it, and set the budget against gradient leakage. If your extraction threat model predates the addition of explanations, redo it.

Decide who gets which explanation. A regulator, an affected individual, an internal auditor and an anonymous API consumer need different things. Serving all four the same format is the failure mode.

Do not treat vendor-supplied explanations as independent assurance. The scaffolding attack means an explanation generated by the party under audit needs weights access or an independent behavioural check behind it.

Negotiate access, not artefacts. If interpretability forms part of your assurance requirement, the term to win in a model contract is the ability to run your own analysis on weights and a training-data sample.

Prefer an interpretable model where one performs. For tabular decisions this removes the attack surface instead of instrumenting it.

Log for reconstruction now. The decision record is the expensive part, and it is needed whichever way the Article 86 commencement question resolves.