Federated Learning Privacy Depends on What the Server Is Allowed to Send

On 10 June 2025 the European Data Protection Supervisor published TechDispatch #1/2025, a technical briefing on federated learning. On the question that decides everything – whether the gradients and weights clients send back can be turned into the training data they came from – the briefing says reconstruction is complicated and works only for a fraction of the training data. In most cases, it says, reconstruction may not be possible at all.

Two years earlier, at IEEE EuroS&P 2023, Franziska Boenisch and five co-authors published a method by which the party running the training recovers more than half of an ImageNet mini-batch of 100 images perfectly, with zero error, in milliseconds. It needs no optimisation objective and no auxiliary dataset, only a server willing to send clients a model whose weights were chosen badly on purpose.

Both statements are accurate. They describe different systems. The EDPS is describing a server that follows the protocol and reads what arrives. Boenisch is describing a server that sends out weights engineered so that what comes back is the data itself. Almost nobody states which one they mean when they say the data never leaves the device.

Federated learning minimises the data a controller collects; it does not make that data anonymous, because a dishonest server operator can recover it from the weights it chooses to send out. Before accepting a federated design as a privacy answer, establish who operates that server and whether the clients can tell what it sent them.

This piece works through what the evidence actually supports. Both the alarming results and the reassuring ones require assumptions that production systems break, and the last section says what a defender should do about it. I have kept the access assumptions attached to every number, because a reconstruction rate quoted without saying whether the server was honest-but-curious or malicious tells you nothing. I am also keeping secure, safe, responsible and trustworthy apart throughout. Almost everything below is security and privacy; where the subject is fairness or data quality instead, I mark it.

How the training loop works, and what leaves the device

Federated learning was named and specified by Brendan McMahan and colleagues at Google in a 2017 AISTATS paper. The algorithm they introduced, FedAvg, remains the default in most federated learning frameworks nine years later.

A server holds a model. Each round it selects some clients and sends them the current weights. Each selected client trains on its own local data for one or more passes and sends back the difference between the model it ended with and the model it started from. The server averages the differences it receives, applies the average to the global model, and starts the next round. Raw data stays where it was generated. Model updates move.

Two structural distinctions matter more than any other for security work, and they are constantly collapsed.

The first is cross-device against cross-silo. In cross-device federated learning, clients are typically consumer endpoints – phones, keyboards, watches – in populations reaching millions. Each holds a small amount of data generated by one person. Participation is intermittent, devices drop out mid-round, and the operator of the server is the platform vendor whose software is also running on the client. In cross-silo federated learning, clients are organisations – hospitals, banks, research institutes – usually numbering in the tens rather than the millions. Each holds a large dataset. Participation is scheduled, identity is contractual, and the party running the server may be a consortium member, a vendor, or a neutral coordinator.

The EDPS briefing draws this distinction and so does most of the academic literature. Almost no security claim respects it. An attack that requires the server to send a tampered model to a targeted client is a different proposition when the client is a phone with no ability to inspect what arrived than when the client is a hospital with a data protection officer and a copy of the expected weights.

The other is horizontal against vertical, a question of which features each client has. Horizontal federated learning means all clients hold the same features for different subjects: every hospital records the same twenty measurements for its own patients. Vertical federated learning means clients hold different features for the same subjects: a bank and a retailer each hold part of the record for the same customers. Vertical designs require an entity-matching step before training and have their own leakage literature. Nearly everything below concerns horizontal federated learning, which is what the attacks discussed here assume.

What actually crosses the wire is a vector with as many entries as the model has trainable parameters. For a small keyboard language model that is a few million numbers. For a federated fine-tune of a modern transformer using low-rank adapters it can be a few hundred thousand. That vector is a function of the client’s private data, the model it was given, and the local training procedure, including whatever randomness that procedure uses; it may also be compressed or sparsified before transmission. The entire privacy question is how much of the input can be recovered from the output, and who gets to choose the model that defines the function.

The two threat models that never meet

The federated learning security literature splits into two halves which do not cite each other’s conclusions, because they assume opposite things.

The poisoning half assumes an honest server and malicious clients. Its adversary controls some fraction of the participating devices and submits crafted updates to degrade the global model or install a backdoor. Its defences run on the server: robust aggregation, anomaly detection, update clipping. This is data poisoning and backdoor implantation with a distributed delivery mechanism.

The privacy half assumes honest clients and a server that is at minimum curious and often actively dishonest. Its adversary is the aggregator, and its defences run on the client or in cryptography: secure aggregation, local noise, trusted execution.

The canonical centrally orchestrated deployment has one server, operated by the party that wants the model, and a client population it does not control. Decentralised variants without a central server exist, and the EDPS briefing covers them, but they are not what most systems run. Both adversaries exist simultaneously. Neither literature models the other’s adversary, and every realism audit published so far assumes a server that follows the protocol.

Three papers have now asked whether attack results transfer to deployment: Shejwalkar and colleagues at IEEE S&P 2022, Valadi and colleagues in a paper revised in February 2026, and Chen and colleagues in March 2026. All three tightened assumptions and found the threat smaller than advertised. Two of the three are about client-side poisoning under an honest server. The third is about gradient inversion under an explicitly honest-but-curious server, and says so in its abstract.

Nothing equivalent has been done for the malicious-server attacks. Carletti and colleagues, discussed below, tested whether four active attacks are detectable at the client, which is an audit of stealth rather than of deployment realism, and it is the closest thing that exists. The Chen paper says outright that it leaves privacy attacks to future work and that the same research-to-practice gap may exist there. So the reassuring results and the alarming results are not in disagreement. They are about different threat models, and the one nobody has audited is the one that produces perfect reconstructions.

What a curious server can read out of the gradients

Start with the weaker adversary: a server that runs FedAvg exactly as specified and studies what arrives.

The foundational result is Deep Leakage from Gradients (Zhu, Liu and Han, NeurIPS 2019), which optimises a dummy input until its gradient matches the observed gradient, and recovers the original training image. Geiping and colleagues improved it at NeurIPS 2020 by matching gradient direction via cosine similarity rather than Euclidean distance, extending recovery to trained networks and deeper architectures.

Reconstruction is not the only leak, and for many deployments it is not the worst one. Melis, Song, De Cristofaro and Shmatikov showed at IEEE S&P 2019 that an adversarial participant infers properties holding for a subset of another participant’s data that are unrelated to the model’s task – for example, when a specific person first appears in the photos used to train a gender classifier. Nasr, Shokri and Houmansadr at the same conference demonstrated passive and active white-box membership inference against federated training, showing that a participant can determine whether a given record was in another participant’s dataset. These are inference attacks against the people in the training data rather than against the model. Both are white-box attacks that read parameters and parameter updates rather than querying a finished model, which is why federated training exposes them at all, and both succeed at far lower fidelity thresholds than image reconstruction does.

Now the important qualification, which the alarmist framing of this literature routinely drops.

Valadi and colleagues at Scaleout Systems and Uppsala University ran gradient inversion across classification and object detection, using current vision architectures at contemporary resolutions rather than the small networks and 32-pixel images the early papers used. Their finding: the modern performance-optimised models they tested resist meaningful visual reconstruction, and many reported successes depend on upper-bound settings – running the model in inference mode, or simplifying the architecture – that do not reflect a real training pipeline. Their conclusion is stated with its access assumption intact: under an honest-but-curious server, high-fidelity image reconstruction is not a critical privacy risk in production-optimised federated systems. The finding belongs to the architectures and training configurations they tested, and their paper is an August 2025 preprint revised in February 2026 rather than a peer-reviewed result.

That paper deserves to be read with its interests visible. Andreas Hellander and Salman Toor are associated with Scaleout Systems, which builds the FEDn federated learning framework. A framework vendor publishing that attacks on federated learning are overstated is not disqualifying, and the experimental work looks careful, but it is the kind of alignment worth naming rather than discovering later. The same courtesy is owed in the other direction: two of the four authors on the Shejwalkar paper arguing that federated learning is robust in practice are Peter Kairouz and Daniel Ramage of Google, one of the largest operators of production federated learning anywhere.

The counterexample to the reassuring result is text. Mohamed Suliman and Douglas Leith at Trinity College Dublin published an attack at ESORICS 2023 specifically targeting the next-word-prediction model in Google’s Gboard. They recover the words a user typed with high accuracy across a range of conditions, and reconstruct word order well enough to recover the sentences. They report that mini-batching and locally added noise do not stop it. Language models leak differently from vision models because the gradient of an embedding layer is sparse and its non-zero rows name the tokens that appeared. That structural fact is not fixed by using a bigger architecture.

So the honest reading of the honest-but-curious case is narrower than either camp states it. Against a passive server, high-fidelity image reconstruction from realistic modern vision training is hard and often fails. Token recovery from language model updates is comparatively easy. Membership and property inference work across both and require far less. Whoever tells you “gradient inversion is solved” or “gradient inversion breaks federated learning” is generalising from one modality to all of them.

What changes when the server stops following the protocol

Everything above assumes the server sends out an honest model. Remove that assumption and the difficulty collapses.

The server chooses the weights it distributes. Clients compute gradients of whatever they are given. A server that wants a specific client’s data does not have to solve an inverse problem – it can hand that client a model engineered so the gradient of its loss is the data, in the clear.

Robbing the Fed (Fowl and colleagues, ICLR 2022) did this by inserting a fully connected “imprint” module early in the architecture, so individual examples fall into separate bins and read straight out of the gradient. Boenisch and colleagues generalised it: their trap weights modify only the values of existing weights, insert no layer, change no architecture, and are constructed to look like ordinary trained parameters. They report perfect recovery of over half of a 100-image ImageNet batch – 57.9% in the reported configuration, which attacks a 3,000-neuron fully connected layer, against 45.7% at 1,000 neurons – at near-zero computational cost, and note that the server can send trap weights to a handful of targeted users in a handful of chosen rounds while every other user receives the real model – which keeps global training working and keeps the anomaly invisible in aggregate metrics.

The line has kept advancing and it has moved toward practicality, not away from it.

• Loki (Zhao and colleagues, IEEE S&P 2024) scales linear-layer leakage to large batches across many clients by manipulating the distributed model.
• Scale-MIA (Shi and colleagues, NDSS 2025) reconstructs inputs through latent space rather than pixel space, and is aimed specifically at federated learning protected by secure aggregation.
• ARES (Gong and colleagues, IEEE S&P 2026) drops the architectural-modification requirement. It formulates recovery from the activations feeding a fully connected layer as a noisy sparse recovery problem, borrowing from compressed sensing, and reconstructs samples from large batches without inserting a module to do the reconstruction.

ARES matters because architectural modification was the detectable part. An inserted layer with many identical weight rows is something a client could notice. A model whose architecture is exactly the one it expects, carrying weights tuned to make an underdetermined inversion solvable, is a much harder thing to spot.

These attacks need no auxiliary data, no expensive optimisation, and no hours of GPU time per recovered example. Most are one-shot: they work in a single round. That changes the economics completely. Optimisation-based inversion under an honest server is a research demonstration with a per-example cost. Analytic inversion under a dishonest server is a data collection pipeline.

Federated fine-tuning does not fix this, and may make it worse

The obvious 2026 use of federated learning is fine-tuning a foundation model on data that cannot be centralised: clinical notes, legal correspondence, proprietary code. The standard efficiency move is parameter-efficient fine-tuning – freeze the pretrained backbone, train small adapter or low-rank modules, and send only those gradients. Practitioners frequently describe this as a privacy improvement, on the reasoning that a few hundred thousand numbers must leak less than a few hundred million.

That reasoning is wrong, and two papers have now shown why.

PEFTLeak (Sami and colleagues, CVPR 2025) reconstructs a large batch of a user’s fine-tuning images with high fidelity from adapter gradients alone. The mechanism is the one above: the server maliciously designs both the frozen pretrained backbone and the adapter parameters it distributes. The frozen backbone is the attack surface, because the client accepts it without training it and therefore without ever examining it.

MineGrad (Sami, Sen and Guler, AISTATS 2026) extends this to low-rank adaptation and to text. It is analytic rather than optimisation-based, needs no adversarial pretraining on public data, and works for both language and vision. Their setup uses 100 users each sending LoRA gradients for a text classification task. The paper makes an operational argument I have not seen stated as plainly elsewhere: federated learning typically runs in the background without active monitoring, so malicious server behaviour is unlikely to be noticed, and because the attack is one-shot and can be run in an early round when accuracy is low anyway, a user who later notices degraded performance is noticing it after the data has already been recovered.

The general principle, and it is the one to carry into any design review: sending fewer numbers is not the same as sending less information. What a gradient leaks depends on how the function that produced it was constructed, and in federated learning the adversary constructs that function.

Secure aggregation and differential privacy, and what each one actually buys

The two controls always offered in response are secure aggregation and differential privacy. They protect against different things and are routinely described as though they were interchangeable.

Secure aggregation

Secure aggregation is a cryptographic protocol, introduced for this purpose by Bonawitz and colleagues at CCS 2017, that lets the server compute the sum of client updates without seeing any individual update. Clients mask their vectors with pairwise secrets that cancel in the sum, with a recovery mechanism for dropouts. It is a real control and it defeats the entire class of attacks that need to attribute an update to a client.

It does not defeat a server willing to break the protocol in other ways. Pasquini, Francati and Ateniese at CCS 2022 showed that a malicious server eludes secure aggregation as if it were not there, by exploiting model inconsistency: sending different models to different clients. Send the target client the real model and send every other client a model whose gradient is guaranteed to be zero, and the “aggregate” the server receives is the target’s update alone, correctly summed, cryptographically intact, and completely deanonymised. The number of participants in the aggregation is irrelevant. Boenisch and colleagues followed with a separate EuroS&P 2023 paper reconstructing individual data points from federated learning hardened with distributed differential privacy and secure aggregation. They report a 95% extraction rate under one condition: the target is placed in a round alongside sybil devices the server controls, and the rate falls as genuine users join the aggregation.

The distinction to hold onto is narrower than “secure aggregation assumes an honest server”, which is what I would have written before reading the protocol carefully. Bonawitz and colleagues prove their construction secure against a malicious aggregator too. What the proof covers is the confidentiality of the values being summed. What it does not cover is whether every client received the same model, whether that model was benign, or whether the other participants were real. Secure aggregation protects the aggregation; it does not authenticate the orchestration around it. That gap is the hole every one of these attacks goes through, and stating it precisely makes the point stronger than overstating it.

Differential privacy

Differential privacy is the control that carries an actual mathematical guarantee, and it is worth being precise about what Google has deployed, because it is the strongest published production example and it is regularly misdescribed.

In 2022 Google announced the first production neural network trained directly on user data with a formal DP guarantee: the Spanish-language Gboard next-word-prediction model, trained with DP-FTRL at ρ = 0.81 zero-concentrated differential privacy, equivalent to (ε = 8.9, δ = 10⁻¹⁰). By 2023 they had deployed more than twenty Gboard language models at ρ between 0.2 and 2, and committed that all future Gboard neural language model launches trained on user data would require a DP guarantee. Those figures come from the arXiv version of the ACL Industry paper; the published ACL text reports more than fifteen models at ρ between 0.3 and 2, with one rather than two using secure aggregation. The commitment on future launches appears in both. By February 2024 they reported the first strong guarantee of ε < 1 for a model trained directly on user data, and described it as the largest known deployment of user-level differential privacy anywhere.

That is genuinely impressive engineering and I would rather more organisations were held to it. Three things about it need saying anyway.

First, epsilon is not a score. Google’s own framing calls ε = 10 reasonable and ε = 1 strong. An ε of 8.9 is a meaningful bound and it is a weak one; it constrains a determined adversary far less than the word “private” implies. A guarantee is only interpretable alongside its privacy unit, its delta, and its composition accounting. A number without those is decoration. I have written before about what differential privacy does and does not promise, and the federated case does not change the arithmetic.

Second, of those twenty-plus models, the arXiv text reports two trained with secure aggregation, and the published version reports one. Absence of a report is not proof of absence, so the precise statement is this: for the models where no separate confidentiality mechanism is reported, DP-FTRL adds its noise server-side, which is the trusted-curator arrangement. The noise in DP-FTRL is added by the server. That is central differential privacy with a trusted curator, and the guarantee it provides is about what the released model reveals, not about what the operator saw on the way there.

Third – and this is the one that generalises – the DP guarantee constrains the released model, not the server’s conduct. A server running trap weights collects data outside the mechanism its epsilon budget describes, so it never has to violate the budget at all.

Local differential privacy, where clients add noise before sending, does address the untrusted server, and it is why the honest answer to “can we just add noise on device” is usually no: the utility cost at meaningful epsilon is severe for anything larger than a small model, and the Suliman and Leith attack on Gboard specifically reports local noise as ineffective at the levels a working keyboard tolerates. Cryptographic alternatives such as homomorphic encryption and secure multi-party computation shift the trust rather than eliminating the cost, and the overheads still rule most of them out at cross-device scale.

The poisoning threat got smaller every time somebody measured it

Turn the threat model around. The server is honest, some clients are not.

The canonical results are strong and well known. How To Backdoor Federated Learning (Bagdasaryan and colleagues, AISTATS 2020) introduced model replacement: a compromised client scales its update so that after averaging it dominates the global model, installing a backdoor while main-task accuracy stays normal. The mechanism is the same one behind neural trojans in a conventional supply chain, with the delivery path changed. Xie and colleagues, in DBA at ICLR 2020, split a trigger into fragments held by different malicious clients, which is both more effective and harder to detect than a single shared trigger. Fang and colleagues at USENIX Security 2020 showed that local model poisoning substantially degrades the four Byzantine-robust aggregators they evaluated. Attack of the Tails (Wang and colleagues, NeurIPS 2020) put backdoors on edge-case inputs at the tail of the distribution, where they survive because nothing benign contests those parameters. That paper also proved something structural: robustness to backdoors implies robustness to adversarial examples, which is an open problem in its own right. Neurotoxin (Zhang and colleagues, ICML 2022) made backdoors durable by writing only into coordinates that benign clients rarely update.

Then people started checking the assumptions.

Shejwalkar, Houmansadr, Kairouz and Ramage enumerated realistic threat models for production federated learning and found that under them, federated learning is robust in practice even with simple, cheap defences – and in several configurations without any defence at all. Their argument is that the literature routinely assumes an implausibly large fraction of compromised clients and adversary capabilities that production systems do not grant.

The Chen and colleagues measurement study, posted to arXiv on 21 March 2026 and not yet peer reviewed, puts arithmetic on it, and the arithmetic is the most useful thing I have read on this question all year. Take a federated system with 100 clients where the server randomly selects 10 per round. For an attacker to get at least four malicious clients selected in a round with 99% probability, they estimate the attacker must have compromised at least 71 of the 100 clients. Recomputed exactly, with sampling without replacement, the threshold is 69; 71 follows from a binomial approximation. Either number makes the point. They give a second figure alongside it: at a 10% join ratio, any one specified client is selected in five consecutive rounds with probability 10⁻⁵. That is a statement about a single client rather than about a whole attack campaign, and it is worth reading as the illustration it is.

Their empirical results follow the same shape. Re-running six backdoor attacks under realistic participation, 1–10% malicious clients, no weight scaling, and joint statistical, device and communication heterogeneity, they find backdoor success becomes strongly dataset-dependent and unstable. On tabular benchmarks several canonical attacks largely stop working. On image and text benchmarks several still succeed, so the risk does not vanish. And attacks that do reach high success often pay for it in main-task accuracy, which makes them operationally visible: their example is DBA on the Purchase100 dataset, which hits a high backdoor success rate at one poison ratio while benign accuracy collapses at the same point.

Their methodological point deserves to outlive the paper. Attack success measured at the final training round overstates practical risk, because it captures transient spikes. They propose measuring the mean and variance of attack success across the last 10% of rounds, alongside the benign accuracy cost. That is the right shape for any attack success rate claim, federated or not, and it is the sort of thing I would want to see before believing a vendor’s poisoning benchmark.

There is a real counter-argument and it should not be buried. MPAF (Cao and Gong, CVPR 2022) accepts that compromising a large fraction of genuine clients is economically implausible, and sidesteps it: inject fake clients instead. Registering synthetic devices costs far less than compromising real ones, and the whole “you would need to own 71 of 100 phones” argument assumes the 100 are real. Sybil resistance at client registration – device attestation, account age, rate limiting – is doing more security work in a cross-device deployment than any aggregation rule, and it is the control most often missing from the threat model entirely. The same reasoning applies to label flipping and GAN-assisted poisoning: the defence is upstream of aggregation.

Robust aggregation is a literature, not a practice

If poisoning is a real if narrower threat, the field has an answer: robust aggregation. Krum and Multi-Krum select updates closest to their neighbours. Coordinate-wise median and trimmed mean discard outliers per parameter. Bulyan composes the two. FLTrust bootstraps trust from a small clean dataset held by the server. FLAME clusters by cosine distance, clips by median norm, and adds calibrated noise. There are dozens more.

Two findings about this body of work should change how you read any claim built on it.

The first is from FLPoison, a February 2025 systematisation that benchmarked 15 attacks against 17 defences across two datasets, two federated algorithms and two heterogeneity levels – 2,040 configurations. Under independent and identically distributed data, several defences work well. Under non-IID data, the authors report that almost no robust defence remains effective against either model poisoning or data poisoning. Only FLTrust holds around 80% average accuracy under non-IID model poisoning, and FLTrust achieves that by assuming the server holds a clean dataset drawn from the same distribution as the clients – which is an assumption that, if you could satisfy it, would already have solved part of your problem.

Non-IID data is not an edge case in federated learning. It is a defining condition of most real deployments: every hospital sees a different case mix, every user types differently. A defence portfolio that works on IID data and fails on non-IID data is a defence portfolio that works everywhere except in deployment.

The cost figures in the same benchmark explain the adoption gap better than any argument about efficacy. Measured per epoch against plain mean aggregation, on LeNet-5 over MNIST under FedSGD, DeepSight costs roughly 1,710,000 times more, FoolsGold about 1,634 times, and DnC about 735 times. Robust statistical aggregators such as Krum, Multi-Krum, median and Bulyan come in at 15 to 53 times. Those ratios belong to that setup rather than to the algorithms in general, and the ordering is what to carry away.

The second finding is the survey of what actually ships. Chen and colleagues tabulated fourteen federated learning platforms from industry and academia and recorded the aggregation and security they support. Google’s TensorFlow Federated, WeBank’s FATE, Huawei’s MindsporeFL, OpenMined’s PySyft, ByteDance’s FedLearner and Baidu’s PaddleFL all list no poisoning-robust security guarantee. IBM Federated Learning offers Krum. Alibaba’s FedScope and Flower offer Bulyan and Krum. Most default to FedAvg. More than half of the industrial platforms implement no secure aggregation; across all fourteen entries it is exactly half.

That is a survey of what frameworks support, not telemetry from running deployments, and it should be read as the former. It still means something specific for how you read papers. Researchers who evaluate a defence against a Multi-Krum deployment are evaluating a system almost nobody operates. An attack evaluation that assumes weight scaling is answering a detection opportunity that secure aggregation removes – and where a platform does not use secure aggregation the server can simply look at the norms.

Where the malicious-server attacks are actually defensible

Put the two halves together and the picture resolves along the cross-device and cross-silo line, which is why I insisted on it earlier.

In cross-silo federated learning, the strongest attacks are defensible. The participants are organisations with security functions. They receive the global model each round as a file. They can hash it, log it, compare what they received against what other participants received, and inspect it for structure. And detection is at least tractable: Carletti and colleagues at the University of Salerno published client-side detection techniques in November 2025 that identify four state-of-the-art active gradient inversion attacks using statistically improbable weight structures and anomalous loss and gradient dynamics, with no modification to the federated learning protocol. Four attacks caught under tested settings is not a solved problem, and an adaptive attacker gets to respond to a published detector. It is still more than the cross-device case has. The model-inconsistency attack in particular is defeated by any mechanism that lets participants confirm they all received the same model.

In cross-device federated learning, they are not. The client is a phone. The user cannot inspect the model, would not know what to look for, and has no channel to compare notes with the other clients selected that round. In the vertically integrated case – Gboard is the obvious one – the server is operated by the vendor whose operating system is also running the client software, so a client-side integrity check is a check the adversary controls. Technical options exist here – attestation, model-consistency protocols, trusted execution on the client, multi-server designs – but each of them either asks the vendor to constrain itself or requires infrastructure that unmanaged consumer endpoints do not have. What is left doing most of the work is governance: contracts, audits, regulatory exposure, and the reputational cost of being caught.

Governance is roughly what protects Gboard users. Publishing epsilon values demonstrates a serious privacy engineering programme; it is not evidence that a malicious aggregator could not step outside the mechanism those values describe. The deterrent is reputational and regulatory, and it should be named as that. But it should be described accurately. A user of a cross-device federated system is trusting the operator’s conduct, with cryptographic and statistical controls that raise the cost of misconduct and do not prevent it.

Which produces a test that is easy to apply and that most federated learning pitches fail: ask what happens to the privacy claim if the server operator is dishonest. If the answer is a control, you have a security property. If the answer is that they would not do that, you have a policy, and the correct thing to do with a policy is to write it into a contract and audit against it. Both are legitimate. Confusing them is not, and it is the specific confusion that lets “the data never leaves the device” do work it cannot support.

What the regulators have said, and where they are behind

European regulators have engaged with federated learning more seriously than with most machine learning security topics, and the resulting documents are better than the field usually gets.

The EDPS TechDispatch is a careful piece of work. It states plainly that it should not be assumed that data exchanged among client devices, or the resulting models, can be treated as anonymous, and that a technical and legal analysis is required in each case. It identifies data quality management as a genuine federated problem, since no party can compare data sources against each other or assess the training set as a whole, which also makes bias in the resulting model harder to detect and harder to correct. It names poisoning, integrity and confidentiality as distinct risks, and lists encryption, secure multi-party computation, trusted execution environments and differential privacy as safeguards. It says no privacy-enhancing technology is a silver bullet.

Its gap is threat model. Its treatment of reconstruction risk – complicated, works for a fraction of the data, in most cases may not be possible – describes an honest-but-curious adversary, and the document does not engage with the malicious-server literature that was three years old and had four IEEE and ACM top-tier papers behind it when the briefing was published in June 2025. A controller reading only that paragraph would conclude that gradient exchange is a low residual risk. Under a server that manipulates the model it distributes, the finding reverses. The document reaches the right instruction – assess case by case – on reasoning that understates the ceiling.

EDPB Opinion 28/2024, adopted 17 December 2024, sets the anonymity test a federated deployment has to meet. An AI model trained on personal data cannot be assumed anonymous; anonymity is assessed case by case, and the threshold is high: the likelihood of extracting personal data directly or probabilistically, and of obtaining it through queries, must be insignificant for every data subject. That standard applies to the global model a federated system produces. It also, on the EDPS reading, applies to the local models and the updates exchanged along the way. Federated learning does not exempt anything from the assessment. It changes where you have to perform it, and it multiplies the number of artefacts you have to perform it on.

On the AI Act, the relevant point for federated designs is a timing one, checked on 6 September 2026. The Digital Omnibus was adopted as Regulation (EU) 2026/1744, published 24 July 2026 and in force from 27 July 2026. It moved high-risk obligations for stand-alone Annex III systems to 2 December 2027 and for AI embedded in regulated products under Annex I to 2 August 2028. The Article 50 transparency duties applied on 2 August 2026 as originally scheduled, and Article 50(2) machine-readable marking applies to systems already on the market from 2 December 2026. Prohibited practices and general-purpose AI obligations did not move.

The deferral matters here because Article 10 data governance – examining training datasets for bias, assessing suitability, documenting provenance – is a Chapter III high-risk obligation, and it is the one federated learning makes structurally hard, for exactly the reason the EDPS identifies: nobody can inspect the union of the training data. A federated clinical model is likely Annex I and now has until August 2028. A federated credit or recruitment model is Annex III and has until December 2027. Anyone still writing to a 2 August 2026 high-risk deadline is working from a calendar that changed in July, and anyone treating December 2027 as distant is misjudging a conformity assessment cycle that runs twelve to eighteen months with notified bodies still being designated.

One more regulatory observation, offered as a question rather than an answer. Federated learning is frequently justified on data minimisation grounds under Article 5(1)(c) GDPR, and the EDPS accepts that argument. But minimisation is about the data a controller collects, and a federated system does not reduce the personal data processed on the device – it relocates the processing. As the EDPS notes in a footnote, personal data still needs to be processed on the client devices, and federated learning does not influence how much training data is used locally. That is a real benefit to the controller’s risk posture and a smaller one to the data subject’s than the framing suggests.

What the deployment evidence actually looks like

One number to calibrate how much of this is live. A systematic review of federated learning in healthcare published in Medical Image Analysis examined 107 studies. Ninety-five used the centralised-server topology. Ten reported real-world deployment in distributed clinical settings. The rest were prototypes or simulations.

Healthcare is the domain where federated learning has the strongest motivation, the most funding and the longest track record. The Federated Tumor Segmentation initiative has published peer-reviewed benchmarking results in Nature Communications, evaluating segmentation algorithms on data distributed across 32 institutions internationally. Owkin, HealthChain and DRAGON have run multi-hospital deployments in Europe. NVIDIA FLARE, Fed-BioMed and OpenFL are real software with real users.

But ten real deployments out of 107 studies is the ratio to hold in mind when someone describes federated learning as the way sensitive AI training gets done. It is a technique with a small production footprint outside a handful of very large consumer deployments and a research-heavy clinical long tail, and most of what is written about its security concerns systems that do not exist yet. That cuts both ways: the attack literature is largely theoretical because the deployments are largely theoretical, and the reassuring measurement studies are calibrated on simulations too.

What I would actually do

One recommendation, with a fork that turns on a fact you can establish in an afternoon.

Treat the aggregation server as the primary adversary in your federated learning threat model, and choose controls according to whether your clients can audit what it sends them.

Cross-silo: audit the model you were sent

If you are running cross-silo federated learning, with a small number of known organisations, you can audit, so audit. Require every participant to verify the global model received each round: hash it, log the hash, and compare hashes across participants out of band, which alone defeats the model-inconsistency attack that breaks secure aggregation. Add the client-side checks from the Carletti detection work for improbable weight structure and anomalous loss dynamics; they are cheap and they need no protocol change. Put model distribution integrity in the consortium agreement as an obligation with an audit right attached. This is the highest-value control available to you and it costs a fraction of what robust aggregation costs.

Cross-device: cryptography, or a commitment you publish

If you are running cross-device federated learning, with consumer endpoints at scale, your clients cannot audit, and no client-side control survives an adversary who ships the client. The only things that carry a guarantee are secure aggregation and a stated user-level differential privacy budget, published with its privacy unit, its delta and its composition accounting. Deploy both, on every model, not on two out of twenty, and publish the epsilon and its accounting rather than the word private. Everything beyond that is a promise about your own conduct, and the correct place for a promise about conduct is a published commitment you can be held to, not a technical claim in a privacy policy.

Four things that apply either way

In rough order of how much they matter relative to what they cost:

1. Put Sybil resistance ahead of robust aggregation. The arithmetic that makes client-side poisoning implausible assumes the clients are real. Device attestation, account age requirements and registration rate limiting defend the assumption that every published robustness argument rests on. Aggregation rules that cost 700 times a mean do not.
2. State the modality in every leakage claim. Text updates leak tokens through sparse embedding gradients in a way vision updates do not leak pixels. A risk assessment that generalises from an image reconstruction result to a clinical notes deployment, or vice versa, is wrong in a direction you cannot predict.
3. Assess membership and property inference separately from reconstruction. They need far less signal, they work under conditions where reconstruction fails, and in most regulated deployments – “was this patient in the study” – they are the disclosure that actually causes the harm.
4. Do not treat parameter-efficient fine-tuning as a privacy control. Adapter and LoRA gradients have been inverted analytically in both vision and text, using a frozen backbone the client never inspects because it never trains it.

And one thing not to do: do not let federated learning substitute for the governance question of who is accountable for the model. Distributing training distributes responsibility, and in every deployment I have looked at, the party operating the server retains all of the capability and acquires none of the accountability that a centralised design would have forced onto it.

What would change my assessment

I have been deliberate about which claims here rest on evidence and which rest on absence of evidence, so here is where I would expect to be wrong.

A realism audit of the malicious-server attacks, done properly. Nobody has run the Shejwalkar or Chen treatment on gradient inversion under an active server: realistic client sampling, production model architectures, heterogeneous devices, and a detection-aware defender. If somebody does that and finds these attacks as brittle in deployment as client-side poisoning turned out to be, the central argument of this piece weakens considerably. The Chen paper explicitly flags this as the next step. Until it exists, the strongest attacks in federated learning remain the least audited, which is not a comfortable place for a field to be.

A documented incident. I could not find a single named, dated case of a federated learning deployment being attacked in the wild – no gradient inversion against a production system by anyone other than researchers with permission, no poisoning of a live federated model. That absence is real and I am reporting it as an absence, not as evidence of safety. Federated deployments are few, most produce models where a backdoor has low value, and detection would be extremely hard, so the base rate should be low and the observation rate lower still. But if this technique is as widely deployed in five years as its advocates expect, that silence will not hold, and the first real case will tell us which half of the literature was closer.

Until then the defensible position is narrower than either camp will like. Federated learning genuinely reduces exposure: it removes the central store of raw training data, which removes the single largest breach target in a conventional pipeline, and that is worth having. It does not make training data unrecoverable, it does not make a model anonymous under GDPR, and it does not protect anyone against the party running the server. Deploy it for what it does. Say what it does not do.