What Still Keeps Frontier Training Concentrated

Large-scale training depends on frequent synchronisation between accelerators. Synchronous data-parallel replicas aggregate gradients at every optimisation step. Frontier runs also use tensor, pipeline and expert parallelism, and each of those adds its own communication dependencies. What they share is a requirement for a fast, predictable interconnect: inside a facility, hundreds of gigabits per second per accelerator with microsecond latency. Run the unmodified algorithm across sites on commodity long-haul links, at sub-gigabit bandwidth and tens of milliseconds, and the accelerators spend most of their time waiting.

Bandwidth is the dominant engineering reason frontier training concentrates, alongside power availability, operational reliability, data pipelines and simple accelerator scarcity. It is not the only reason, and it is not an absolute one.

Over premium fibre it is already solved. NVIDIA reported training Nemotron-4 340B across two data centres roughly 1,000 kilometres apart, 1,500 GPUs at each site, at over 96% of single-site throughput, by overlapping inter-site and intra-site communication rather than by reducing it. Epoch reports Google split Gemini 2.5 training across several nearby buildings. Meta has described assembling a single large cluster from multiple production data centres. Frontier training is not confined to one building and has not been for some time.

This matters because every argument about attacking AI compute depends on the target being large, identifiable and singular. The interesting question is no longer whether a training run can leave the building. It demonstrably can. How far it can spread before synchronisation costs and frontier economics pull it back is not established. The distance has increased every year and nobody has published where it stops.

There is a second constraint underneath the first, and it pushes the same way. Accelerators need power, at densities that most grids cannot deliver on short notice. Operators therefore site clusters where spare generation and interconnection capacity exist, which is a small set of locations. Bandwidth says put the machines together; power says put them where the electricity is. Bandwidth and power both point at a handful of very large facilities, and they reinforce each other rather than trading off.

China’s Eastern Data, Western Computing programme, formally launched in February 2022, established eight national computing hub nodes and ten data-centre clusters, four hubs in the resource-rich west and four in the eastern population centres. The programme moves suitable workloads west because the eastern regions lack land and energy. It was built for energy economics and regional development rather than survivability, and it is one of the clearest large-scale examples of a state deliberately shaping the geography of national compute.

The split that never closed

The survivability question was answered once already, for a different layer.

Paul Baran’s RAND memoranda in 1964 were commissioned to work out how command and control could survive a nuclear first strike. His answer was a redundant mesh with no central node, distributed routing, and messages broken into blocks that find their own way. That became packet switching, and eventually the internet.

What it solved was communication, and it was one of several independent origins of packet switching alongside Donald Davies’ work at Britain’s National Physical Laboratory.

Designers took computation the other way, and most retellings get SAGE wrong. SAGE was not one vault: at full deployment it ran twenty-four Direction Centers across the United States, each holding its own dual AN/FSQ-7. That is geographically distributed computation, but into fixed regional processing centres, each doing its own work. It never produced fungible compute that could redistribute one tightly coupled workload around a destroyed site. Cheyenne Mountain, operational from 1966, was hardened command and correlation rather than where SAGE’s computing went.

For sixty years nobody closed the split. Communication learned to route around damage. Computers stayed large, hardened and few, and everything we know about surviving as a distributed system, we know about moving bits rather than about processing them.

Grid computing closed part of that gap and it is worth being precise about which part. Ian Foster and Carl Kesselman’s work in the late 1990s, and the Globus toolkit that came out of it, produced real geographically distributed production computing. The Worldwide LHC Computing Grid still runs on that model, linking around 160 computing centres across more than forty countries, with CERN as tier zero, fourteen national tier-one centres on dedicated optical links, and a tier-two layer beneath them.

It works because collider analysis decomposes into enormous numbers of largely independent jobs. Data locality and scheduling still matter, but there is no global synchronisation step, so there is nothing for latency to poison.

Conventional large-scale model training is the opposite kind of workload. Groups of accelerators need each other’s output before the next step begins. That structural difference is why thirty years of grid computing did not transfer.

What the low-communication research actually shows

It is being made to transfer now, and the claim needs stating carefully because the coverage of it does not.

DiLoCo, published by Arthur Douillard and colleagues at Google DeepMind in November 2023, restructures the problem. Each worker runs a large number of local optimisation steps with AdamW, then the workers synchronise through an outer optimiser using Nesterov momentum. Communication happens every few hundred steps instead of every step. The reported result is that eight workers match fully synchronous optimisation while communicating five hundred times less.

Whoever quotes that number usually drops two conditions. The experiments ran on C4 at model sizes from 60 to 400 million parameters, with the headline result at 150 million. And it is an arXiv preprint later presented at an ICML 2024 workshop, which is real review but thinner than the standing the figure usually gets.

The scale has moved a long way since, and the ladder is worth setting out because most accounts stop at the first rung. OpenDiLoCo reproduced the 150-million-parameter result, then ran the genuinely geographically distributed experiment at 1.1 billion parameters across four workers in Canada, Finland and two US states, on links measured between 127 and 935 megabits per second. Prime Intellect then trained a 10-billion-parameter model on a trillion tokens. INTELLECT-1 ran on up to fourteen concurrent nodes spread across five countries on three continents. Thirty independent providers joined and left mid-run. Compute utilisation held between 83 and 96 percent at roughly a four-hundred-fold bandwidth reduction. Then in April 2026 Google DeepMind published Decoupled DiLoCo, which combines the low-communication approach with asynchronous scheduling and fault-isolated islands of compute. They trained a 12-billion-parameter Gemma 4 model across four separate US regions over wide-area links of 2 to 5 Gbps, bandwidth they describe as achievable on existing connectivity between facilities rather than requiring custom networking. Benchmark accuracy came out at 64.1% against 64.4% for the conventional baseline, and the run completed more than twenty times faster than synchronous methods.

DeepMind calls it practical for production-level, fully distributed pre-training, and that phrase is the one that matters.

For the arithmetic, since people inflate this comparison: 10 billion is about 67 times the 150-million headline and 25 times the paper’s largest run, so roughly an order and a half rather than the two orders it is usually called. Twelve billion is still well below frontier training budgets, and parameter count is a poor proxy in any case now that the EU AI Act uses cumulative training compute above 10^25 FLOP as its systemic-risk threshold. This is no longer a small-scale research curiosity, and it is the organisation with the most frontier training experience in the world saying the constraint is an architectural choice.

Nobody publishes military survivability as the rationale. It is economic and operational: compute is fragmented and accelerators are scarce. Bandwidth is expensive, and power is the binding constraint on new capacity. A method that lets you train across three mediocre clusters instead of building one excellent one is worth a great deal of money.

What would settle the question is a complete pretraining run at frontier compute budgets, spread across sites on ordinary connectivity, with loss curves and link budgets published. That does not exist. NVIDIA’s 340-billion-parameter two-site run is closest on scale and furthest from ordinary connectivity: premium long-haul fibre, conventional synchronisation made to work rather than avoided. The low-communication line, which is the one that would work over commodity links, tops out publicly around twelve billion.

Two approaches are converging on the same destination from opposite directions and neither has arrived. Reading that as an accomplished fact is an error. So is dismissing it.

There is also a reason a lab might not publish if it did succeed, and I offer this as speculation rather than reporting: confirming that your training is no longer concentrated is useful information to exactly the people you would least want holding it.

The distinction that gets conflated

Federated learning and distributed training produce the same topology and answer different questions. People conflate them constantly.

Federated learning exists to leave source data under separate administrative control. The motivation is privacy or regulation: patient records never leave the institution that holds them. Distributed low-communication training exists to aggregate fragmented compute. The motivation is capacity.

DiLoCo’s authors describe it as a variant of federated averaging, which is why the confusion is understandable, and the algorithmic families genuinely overlap. What separates them is not the threat catalogue: both face inference attacks against training data and both face poisoning, backdoor and availability threats. What separates them is the trust assumption. Federated participants are mutually untrusting by design, which is the whole reason the data does not move. Distributed training workers usually belong to one operator, and the security question is whether that holds. INTELLECT-1 ran with thirty independent providers joining and leaving, which is a different trust arrangement wearing the same topology.

That last one is the interesting one for this beat. DiLoCo’s authors report the method tolerates workers dropping out during training and new ones joining. Decoupled DiLoCo goes further: DeepMind used chaos engineering to induce hardware failures and reported that training continued after entire learner units were lost, then reintegrated them when they came back. In simulation at 1.2 million chips it held 88% goodput where standard data-parallel training managed 27%.

Those properties were engineered for hardware failure, preemptible capacity and stranded compute. They also describe, precisely, what a training run would need in order to survive losing a site, which is not what they were built for.

What this changes

Three things, in descending order of confidence.

Geographic concentration is an engineering trade-off rather than a law of model training, and the feasible distance between cooperating compute increases every year. Any strategic argument that depends on frontier training being observable at a small number of large sites has a shelf life, and the people writing those arguments should say what happens when it expires.

The evidence still does not reach frontier compute budgets, and anyone claiming a settled result is overreading. Both are true at once, and holding both is the discipline here.

There is a third-order effect worth raising as a question rather than a finding. Compute verification gets harder when a regulated training run can be assembled from capacity spread across providers and jurisdictions. Five sites would still leave power draw, chip procurement, cloud billing, network and scheduler evidence, so the signature does not disappear. What degrades is the inference that a single very large run implies a single building. A monitoring regime built to find that building would have to aggregate evidence across sites instead. That is a problem for compute thresholds in any agreement, and for the intelligence assessment a strike decision would require.

And for anyone actually operating this infrastructure, the useful consequence is not geopolitical. It is that the resilience properties being built into these methods for commercial reasons are the same properties you would want if you took physical risk to your compute seriously, and almost nobody is evaluating them that way.

There is also a harder question underneath, which is whether destroying training capacity destroys anything that matters once a model already exists. It does not, and that is the objection the strike proposals have not answered.

Sources checked 5 September 2026. Model sizes and communication ratios verified against the arXiv papers rather than secondary coverage.