Cerebras Runs a 3-Trillion-Parameter Model on 59 Wafers

On October 22, Cerebras published the first detailed public technical document on how inference works for models with more than one trillion parameters on its wafer-scale chips. The article uses Moonshot AI's Kimi K3 as an example — an open model with approximately 3 trillion parameters, 69 linear attention layers and 24 dense layers, distributed across 59 WSE-3T wafers to generate between 1,000 and 2,000+ tokens per second.

The publication is significant not for being theoretical — Cerebras had already mentioned Kimi K2.6 in May, with performance around 1,000 tokens per second — but for revealing concrete engineering details that were previously kept under wraps. It is the equivalent, in the AI infrastructure field, to the first public paper of a distributed training data center: less marketing, more specifications that any engineer can analyze.

Architecture inside out

Kimi K3 has 2.8 trillion parameters. If each parameter is stored at 4 bits (int4 quantization), the raw weight payload occupies about 1.4 terabytes. Each WSE-3T wafer has 44 gigabytes of on-chip SRAM — enough for individual layers, but far from what is needed for the entire model.

Cerebras' solution involves pipeline parallelism. The model's 93 layers are mapped to regions of separate wafers. The 69 linear attention layers require 35 wafers, and the 24 dense layers need 24 wafers — totaling 59. Each wafer executes its slice of the network and sends only intermediate activations to the next wafer, using RDMA over Ethernet with approximately 2 microseconds latency per hop.

What differentiates Cerebras' approach from conventional GPU use is fundamental: weights never leave on-chip SRAM. In GPUs with HBM (High Bandwidth Memory), token generation is limited by the bandwidth between the chip and external memory. In the WSE-3T, internal bandwidth reaches 43.2 petabytes per second per CS-4 rack (three wafers), eliminating this bottleneck structurally.

Energy consumption: the other side of the coin

Each WSE-3T wafer consumes about 20 kilowatts. With 59 wafers in operation, total consumption reaches approximately 1.18 megawatts — equivalent to a small hydroelectric power plant. To generate about 1,500 tokens per second with 190 concurrent requests, Cerebras estimates consumption of 4.1 joules per token. The company considers this "reasonable" for a model of this magnitude, but it reveals the raw truth of scale costs: models of this size are, literally, power plants.

The comparison with GPU is revealing. In a typical multi-GPU configuration (for example, 8x AMD Instinct MI355X with 288 GiB of HBM3E each), Kimi K3 inference is feasible but limited by constant weight swapping between GPUs via interconnect. Cerebras, on the other hand, keeps everything on-chip — and pays the price in kilowatts per wafer.

The market forming around wafer-scale

The technical publication does not come in isolation. In the two weeks prior, two distinct ecosystems had already announced integration with Cerebras:

Gimlet Cloud announced, on September 28, a partnership with Cerebras to offer disaggregated inference at up to 3,000 tokens per second. The initial data center is expected by the end of 2026, with 100 megawatts of capacity — a position that Futurum Group places behind only OpenAI in the Cerebras capacity queue. Access to CS-4 (next generation) is expected for 2027.

General Compute closed a multi-year deal the following day, with $400 million in debt financing from Upper90 to build dedicated infrastructure for agentic coding inference, with planned availability for Q1 2027.

Both cases follow the same pattern of disaggregation: prefill (computationally intensive prompt processing) runs on conventional GPUs, while decode (token generation) migrates to Cerebras wafers, whose SRAM moves data faster than any HBM on the market. The disadvantage is that wafer-scale infrastructure does not exist in public clouds — only in niches of specialized players like Gimlet Labs and General Compute.

What to watch

Cerebras' publication confirms an increasingly clear trend: inference of frontier models is separating from training infrastructure. While NVIDIA, AMD, and Google continue investing billions in chips oriented toward distributed training (which are, after all, the ecosystem's most expensive bottleneck), inference at scale is developing its own architectures — specialized, energetically intensive, and increasingly focused on latency as a performance metric.

The question to watch is whether this wafer-scale inference model generalizes or remains a niche solution for the few players with the capacity to acquire hardware. The CS-4, available in 2027, promises to double the CS-3's bandwidth. But perhaps the more important question is this: as models grow from 3 to 10 trillion parameters, how many megawatts will each inference data center need to survive?

Sources: Cerebras, The Intel Brief, Futurum Group

✓ Independent sources cross-checked and verified before publishing