Homomorphic Encryption for ML: What the Overhead Buys

Correction note, 6 September 2026: this article previously described CryptoNets as demonstrating that neural networks can be trained on ciphertext. CryptoNets is an inference result. The article also stated the technique’s trade-offs as largely resolved without giving a single performance figure. Both are corrected below.

---

In February 2026 a preprint called Cachemir reported fully homomorphic Llama-3-8B decoding with a scheme for managing the KV cache across variable sequence lengths. On a GPU it generated one output token in under 100 seconds. On CPU the same system ran 48.83 times faster than MOAI and 67.16 times faster than THOR, the two prior systems it measured itself against.

One token. Under a hundred seconds. On a GPU. And that is the number after the speedup.

The claim you will hear at conferences is that homomorphic encryption has arrived and private AI inference is now practical. Both are true as they are usually meant. The distance between them decides whether FHE fits a given workload.

Homomorphic encryption lets an untrusted party compute on data it cannot read. The overhead is severe enough that the set of machine learning workloads where that trade is correct is small, specific and worth naming precisely.

What it gives you

Every other privacy technique in this category makes a trade you can describe without cryptography. Masking changes what a query returns. Differential privacy adds calibrated noise and bounds what any record contributes. Access control keeps the wrong people away from the data.

Homomorphic encryption removes the requirement to trust the party doing the computing. The server receives ciphertext, computes on it without decrypting, returns an encrypted result, and under the scheme’s security assumptions learns nothing about the plaintext inputs or intermediates, because it never holds a key. Secure multi-party computation reaches a related place by a different route, paying in interaction and network traffic where FHE pays in compute. What is distinctive about FHE is that a client can hand ciphertext to a single untrusted party which evaluates it without talking back. That property is what the overhead described below buys.

One deployment shape makes the guarantee bidirectional. A model owner runs inference for a client. The client’s input stays encrypted, so the model owner sees only ciphertext, and the weights stay on the server, so the client never sees the model. Two parties who will not show each other anything can still transact.

The overhead, with numbers

The old version of this article said the trade-offs had been mitigated in part and left it there. The figures are better than that and worse than the marketing.

There is no useful universal multiplier, and anyone offering one is selling something. Around ten thousand times plaintext is the common order-of-magnitude reference for general-purpose CPU execution. Application cost then swings by orders of magnitude with the scheme, the circuit depth, the packing strategy, the bootstrapping placement, the batch size and the hardware. Accelerators remove one or two orders of magnitude from FHE execution time, which is a different statement from the resulting workload sitting one or two orders of magnitude from plaintext. Those are different denominators, and vendor material conflates them routinely.

Named results are more useful than a range. HyPHEN brought ResNet-18 on ImageNet to 14.7 seconds on a single A100. FHEON reports 13 seconds for LeNet-5 on MNIST and 403 seconds for ResNet-20 on CIFAR-10 on a consumer CPU. Cachemir’s Llama-3-8B token is the 96.6 seconds above. Duality’s own write-up puts encrypted logistic regression over ten thousand records at two to ten seconds with optimised CKKS and batching, which is a vendor figure and should be read as one.

There is one claim-evidence gap here worth naming, because it circulates widely. The minutes-per-token figures quoted for encrypted LLM inference generally come from secure multi-party computation research rather than from homomorphic encryption. They are separate technologies with different trust and network assumptions, and citing an MPC latency figure as an FHE result overstates where FHE is by a wide margin. When a vendor gives you a number for private LLM inference, the first question is which technique produced it.

Which scheme, and why it matters

Three scheme families dominate and they are not interchangeable.

CKKS computes on approximate real numbers with heavy SIMD packing, which suits neural network linear algebra, and it is what HyPHEN, FHEON and Cachemir all use. It is not the only way to run a network: Zama’s Concrete ML builds them on TFHE. Approximate is not a caveat you can ignore, because it makes precision loss a design parameter.

TFHE computes on bits and small integers with fast programmable bootstrapping, which makes it the right tool for comparisons, branching and exact logic. Anything with a decision in it tends to want TFHE.

BFV and BGV handle exact integer arithmetic, useful where a lookup has to be right rather than close.

Some systems combine them through scheme switching; plenty of serious ones stay inside a single scheme to keep the engineering tractable. Either way the choice comes before everything else. The library landscape has consolidated enough to name: OpenFHE implements BFV, BGV, CKKS and the TFHE family, Zama’s stack compiles Python models to TFHE execution, Google maintains the HEIR compiler, and Apple has open-sourced a BFV implementation in Swift.

What it does not protect

Two limits get elided, and both matter more than the performance figures.

The output still leaks. Homomorphic encryption protects the input during the computation. It says nothing about what the decrypted result reveals. A client that sends encrypted records gets predictions back in ciphertext, and those predictions still expose whatever the model infers, so a client with enough queries can perform model extraction or membership inference against a model it cannot see.

Two different problems hide in that sentence and they have different answers. Differential privacy can bound how much any individual training record influences a released model, which is what addresses membership inference and reconstruction. NIST is explicit that it does not guarantee against model extraction, because DP protects the training data rather than the model. Rate limits, query accounting and output restriction reduce extraction risk without formally bounding it.

It assumes the computation is the one you asked for. Standard FHE gives confidentiality, not integrity of computation. A malicious server can return the encrypted result of a different function and the client cannot tell without additional machinery. Verifiable computation is a separate and much less mature line of work.

So the honest position is that FHE composes with the other techniques rather than replacing them. A deployment that needs confidentiality from the compute provider and resistance to inference from the client needs both FHE and differential privacy, and paying for one does not buy the other.

Where it fits today

Four shapes, and the pattern connecting them is that latency is not the binding constraint.

Batch analytics on data you cannot see. Encrypted computation over a partner’s records, results returned encrypted, run overnight. The overhead is irrelevant when the alternative is not doing the analysis.

Small models and classical statistics. Logistic regression, small trees, low-depth networks. Taubert, Skuta and Loruenser benchmarked FHE against secure multi-party computation for ML inference in Applied Sciences in June 2026, and found FHE easier to deploy and better performing on regression and simple models, while MPC scaled better to complex architectures such as CNNs and degraded sharply under network latency. That compares particular frameworks on one set of hardware rather than proving a theorem, and it is still the most useful decision rule available.

Secure aggregation. Combining model updates from multiple parties without the aggregator seeing any individual contribution is a low-depth arithmetic problem, which is the case FHE handles well. Additive and threshold schemes and purpose-built protocols often do it more cheaply, so FHE is a workable option here and not a requirement.

Workflows where the compute provider is the adversary in the threat model. Health and financial data processed by a third party under a requirement that the third party must not be able to read it. That requirement is usually contractual or architectural, and rarely statutory. Where it holds, a large slowdown on a batch job is an acceptable price.

It does not fit interactive inference on a large model. Cachemir’s number is the reason. Progress since 2016 has been fast enough that the answer may change, and too slow for any roadmap to depend on that happening soon.

The quantum question, briefly

The article’s original framing was loose here and it is worth stating correctly. CKKS, BFV and TFHE take their security from LWE and ring-LWE assumptions. Those assumptions are related to the module-lattice assumptions behind NIST’s post-quantum key establishment and signature standards without being identical to them, and NIST has not standardised these schemes. There is no known Shor-style attack against that family, so FHE built on it does not face the cryptanalytic threat that RSA and elliptic-curve cryptography do, given appropriate parameters. That property is genuine and often overlooked. It is still not a reason to adopt FHE, which you would deploy for the confidentiality guarantee or not at all. The migration argument belongs on PostQuantum.com, which covers that subject.

What to do on Monday

Establish whether the compute provider is in your threat model. This is the only question that determines whether FHE is relevant. If your threat model trusts the compute environment while plaintext is being processed, FHE solves a problem you do not have. Make that decision explicitly: encrypting data in storage and in transit protects it in those two states and does nothing for data in use, which stays exposed to a compromised host, hypervisor or administrator.

If it is, cost the batch version before the interactive one. Most requirements that arrive framed as private inference turn out to be periodic analysis with a latency budget measured in hours. That version is affordable today.

Demand the technique behind any performance number. FHE, MPC, trusted execution environments and confidential GPUs are four answers with four different trust models. Their benchmarks are routinely quoted interchangeably.

Do not let it substitute for the model-level controls. Encrypted input plus an unprotected model still leaks through outputs, and the attacks in this category work on exactly that channel.

Homomorphic encryption was a theoretical question for thirty years and has been a systems-engineering problem since Gentry’s 2009 construction, and it has moved fast. It is now genuinely deployable for a specific class of workload, and nowhere near the interactive inference most people picture when they hear private AI. The earlier version of this article said only the first of those two things.