Feature Attribution Attacks: When Explanations Become the Vulnerability

Four methods account for most explainable AI in production. LIME, or Local Interpretable Model-agnostic Explanations, fits a simple linear model to the target model’s answers on perturbed copies of an input. SHAP, or SHapley Additive exPlanations, gives each feature a Shapley value from cooperative game theory, and its default implementation, Kernel SHAP, arrives there the same way LIME does, by querying the model with modified inputs and watching the output move.

Integrated Gradients integrates the model’s gradient along a straight path from a baseline input to the real one. Grad-CAM, a class activation map, weights the activations of a convolutional layer by the gradients flowing into it and upsamples the result into a heatmap over the image. All four return the same shape of answer: a relevance score per input feature, for one specific prediction.

The explanation is computed from the model, by querying it or by differentiating it, so a change to the model changes what its explanations say. On a hosted service the same endpoint returns both, so every explanation is a second output describing the model’s internals.

Auditors, regulators and internal risk functions have begun treating these outputs as evidence. Article 13 of the EU AI Act requires that a high-risk system be transparent enough for a deployer to interpret its output and use it appropriately. It prescribes no technique, and feature attribution is what the market reached for.

The bottom line for a security team: an explanation is not independent evidence about a model, because it is generated by the same system that is under review. Ghorbani, Abid and Zou corrupted what an explanation shows in 2019. Milli, Schmidt, Dragan and Hardt reconstructed a model from its explanations that same year.

Manipulating what explanations show

Ghorbani, Abid and Zou showed at AAAI 2019 that saliency maps are fragile. A perturbation invisible to the human eye moves the map onto entirely different features while the model’s predicted label stays the same. They tested vanilla gradients, DeepLIFT and Integrated Gradients on the standard image benchmarks ImageNet and CIFAR-10, an attack surface covered in more depth in saliency attacks.

Their own illustration is medical: a pathology image that keeps its malignant classification while the map sends the clinician to a different region. The prediction is identical in both cases, so it cannot settle which map the clinician should act on.

Dombrowski, Alber and colleagues went further at NeurIPS 2019, forcing the explanation to match an arbitrary target map, including one that spelled out a sentence, while the network’s output stayed approximately constant across all class probabilities.

The perturbed input remains on the hypersurface of constant network output, and the change in a gradient explanation is bounded by the principal curvatures of that surface and by the geodesic distance travelled along it. High curvature of the decision function is the culprit, not a flaw in any one method, which is why the attack reproduced across six methods including LRP and Pattern Attribution.

They also proposed a defence. Computing the explanation on a network whose ReLUs have been swapped for softplus units lowers the curvature, and SmoothGrad achieves something similar at higher cost. Noppel’s team defeated that defence four years later.

Slack, Hilgard, Jia, Singh and Lakkaraju turned fragility into a tool at AIES 2020. LIME and Kernel SHAP both probe a model with perturbed copies of a record. Those probes come from a distribution that a classifier can learn to distinguish from real data.

Slack’s team trained exactly that detector, then wrapped a deliberately racist or sexist classifier in a scaffold: real records go to the biased model, probe records go to an innocuous one. On COMPAS and Communities and Crime, LIME dropped the sensitive attribute out of its top-ranked feature for 100% of held-out points, and for 91% on German Credit. SHAP proved harder and still fell, at 84%, 100% and 85% across the three datasets.

The adversary is the party being audited, who controls the classifier and hands the auditor black-box access to it. A compliance process that accepts a SHAP report as proof that a lending model does not discriminate can be defeated by the organisation producing the report.

Slack’s team called their construction a scaffolding attack. The term fairwashing came from Aïvodji and colleagues at ICML 2019, whose LaundryML produced fair-looking rule lists that faithfully approximated an unfair black box.

Explanation-aware backdoors

Noppel, Peter and Wressnegger published the most operationally alarming result in this line at IEEE S&P 2023. They fine-tune a model so that a trigger in the input flips the prediction and forces a chosen explanation at the same time. The model manipulation is a backdoor in the ordinary sense, with a second objective bolted on.

The paper covers three attack modes: force a specific explanation pattern, throw what the authors call a red herring by pointing the analyst at innocuous features while the prediction flips, or fully disguise the attack by returning the explanation the clean model would have produced. They evaluated three families of white-box method – vanilla gradients, Grad-CAM, and the propagation-based Relevance-CAM of Lee and colleagues – on CIFAR-10 and GTSRB, using a ResNet20 whose clean accuracy stayed within a few points of the unmodified 91.9%.

Against DREBIN Android malware detection, with rarely-occurring URLs as the trigger, the backdoored model held an F1 of 0.672 against the original 0.679 and classified every triggered malware sample as benign. It also replaced the analyst’s top ten features with the ten most common goodware features, at an intersection score of 0.999. The analyst sees a malware sample scored as goodware and explained by the features of goodware.

Two defences that use explanations to catch backdoors fail here. SentiNet locates a trigger by masking whatever Grad-CAM marks as relevant. Overlap between that mask and the real patch drops from 0.706 for a traditional backdoor to 0.001 for this one. Februus cuts out the highlighted patch and inpaints it, which takes a traditional backdoor’s attack success rate from 100% to 6.3% and leaves the explanation-aware version at 99.9%.

Noppel’s team also tested Dombrowski’s defence. SmoothGrad reduced the attack’s effectiveness slightly. An attacker who folds SmoothGrad into the optimisation restores it completely.

Explanations as an extraction interface

The attacks above corrupt what an explanation shows. A second body of work shows the explanation interface leaking the model itself. It reaches beyond feature attribution: counterfactual explanations, which tell a user what would have to change for a different outcome, produce the strongest extraction results of the set.

Milli, Schmidt, Dragan and Hardt proved at FAT\* 2019 that an attacker who can query a model’s input gradients at chosen points can learn a two-layer ReLU network, with a query count independent of the input dimension and near-optimal in model size. Their heuristics reconstructed real models orders of magnitude more query-efficiently than attacks using the prediction interface alone, which is the baseline case covered in model stealing.

That result is about input gradients. It does not transfer to Shapley values, and citing it as a result about SHAP is a common error.

Wang, Qian and Miao showed at FAccT 2022 that a counterfactual is by construction a point close to the decision boundary. Their DualCF strategy takes each one and asks the API to explain it in turn, which yields pairs of points straddling the boundary and a high-fidelity substitute from far fewer queries.

Oksuz, Halimi and Ayday exploited LIME and SHAP in PoPETs 2024 to extract decision trees, logistic regression, naive Bayes and k-nearest neighbour models across six datasets, with fewer queries than prior attacks and resilience to the countermeasures proposed against them. Miura, Shibahara and Yanai went data-free at SecTL 2024: querying a gradient-based explainer, their generative approach cloned a CIFAR-10 model to 92.3% accuracy where the prediction-only baseline reached 80.6%.

Ezzeddine, Giordano and Ayoub published the most recent result in May 2026, distilling a substitute model from counterfactual explanations with no knowledge of the training distribution.

On the Give Me Some Credit dataset, 80% agreement with the target model took 50 queries when counterfactuals were returned and over 1,000 when they were not. Adding differential privacy to the counterfactual generator cost the attacker 10 to 20 points of agreement and degraded the counterfactuals as recourse advice. Their conclusion is that differential privacy narrows the attack surface without closing it, and needs pairing with rate limiting and query auditing.

Across gradient explanations, LIME and SHAP outputs and counterfactuals, each explanation returned tells an attacker more about the model than the prediction it accompanies.

What the defender faces

Three different adversaries appear in the work above. Ghorbani and Dombrowski assume an attacker who can only submit inputs, which is the adversarial example threat model applied to the explanation instead of the prediction.

Slack and Noppel assume an attacker who controls the model. In an audit that is the organisation under review. In a supply chain it is whoever trained or fine-tuned the weights you downloaded. Milli, Wang, Oksuz, Miura and Ezzeddine assume an outsider with query access and nothing else. No single control answers all three, which is why a programme built against the outside attacker alone misses the other two.

The EU AI Act splits along a line that leaves this uncovered. Article 13 asks for interpretability of the output and sets no standard for whether the interpretation resists manipulation. Article 15 does impose cybersecurity duties on high-risk systems, and it names the attacks it has in mind: poisoning of the training data or of pre-trained components, adversarial examples and model evasion, confidentiality attacks, model flaws. Manipulation of an explanation is absent from that list, and so is extraction through an explanation interface.

Both articles apply from 2 December 2027 for Annex III high-risk systems and from 2 August 2028 for Annex I systems, after Regulation (EU) 2026/1744 moved those dates. Checked 5 September 2026.

Defences exist, and none of them survives an adaptive attacker on current evidence. Baniecki and Biecek catalogue the families in their 2024 survey: smoothing, aggregation, ensembles, constrained training. Dombrowski’s smoothing falls to Noppel’s adaptive optimisation. An ensemble of explanation methods only helps until the attacker optimises against all of them, which Noppel’s team did for three methods in one fine-tuning run. Differential privacy on counterfactuals costs explanation quality and reduces the extraction advantage without removing it.

That assessment is mine and not a finding of the survey. I would revise it for any defence tested against an attacker who knows the defence is there.

Three things follow for a defender this quarter. First, an explanation report is evidence about whoever produced it, so establish who controlled the model, the explanation pipeline and the data behind both before the report enters an assurance file.

Second, an explanation endpoint carries more information per query than a prediction endpoint, so give it its own rate limits, query budgets and logging, and do not enable it by default for every API consumer. Third, any control that reads explanations to detect backdoors or bias needs a second detection path that does not read the same explanation, because SentiNet and Februus both failed for exactly that reason. The explainability frameworks an organisation adopts for assurance should be assessed on that basis, not on the quality of the picture they produce.

Explanations remain useful for debugging, and counterfactuals are returned in the first place because a person denied a loan is owed an account of what would change the answer. Treating either one as a security control is the mistake.