Google announced two chips at Cloud Next this week instead of one. The eighth generation TPU splits into TPU 8t for training and TPU 8i for inference — the first time Google has divided its flagship accelerator line by workload type. Both ship later this year, with general availability "later this year" and a request-for-information list open now.
The split matters less than the memory numbers on the inference part.
TPU 8i: 288 GB HBM, 384 MB On-Chip SRAM
The inference chip pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM — three times the SRAM of the previous generation (Ironwood). Google's stated reason is that the KV cache footprint of reasoning models at production scale was outgrowing on-chip capacity, forcing processors to idle while they wait on memory.
Alongside the memory change:
- Interconnect (ICI) bandwidth doubled to 19.2 Tb/s for Mixture-of-Experts models
- A new Boardfly topology cuts maximum network diameter by more than 50%
- An on-chip Collectives Acceleration Engine (CAE) offloads global operations, reducing on-chip latency by up to 5x
- Google doubled the physical CPU hosts per server and moved to its own Axion Arm-based CPUs, with NUMA isolation
The headline economic claim: 80% better performance-per-dollar than the previous generation, letting customers serve "nearly twice the customer volume at the same cost."
That is a vendor figure, not an independently verified one. Treat it as a pricing signal, not a benchmark. But unlike the usual "up to N× faster" framing, it is anchored to a specific capacity claim (twice the serving volume at the same cost) that is falsifiable once GA pricing appears.
TPU 8t: 9,600 Chips, 2 PB Shared HBM, 121 ExaFlops
The training chip goes the other direction — scale over per-chip memory:
| Spec | TPU 8t figure |
|---|---|
| Chips per superpod | 9,600 |
| Shared high-bandwidth memory | 2 petabytes |
| Pod compute | 121 ExaFlops |
| Interchip bandwidth | 2× previous generation |
| Storage access | 10× faster, with TPUDirect |
| Target goodput | over 97% |
| Compute per pod | nearly 3× previous generation |
Google claims near-linear scaling to up to a million chips in a single logical cluster via the new Virgo Network fabric, JAX, and Pathways. Reliability work includes real-time telemetry across tens of thousands of chips, automatic rerouting around faulty ICI links without interrupting a job, and Optical Circuit Switching that reconfigures around failures without human intervention.
At frontier training scale, Google notes, every percentage point of goodput translates into days of active training time. That is the real number in the 8t column — not the ExaFlops.
What Actually Changes for You
If you rent GPU capacity, the relevant part is the 8i line. An 80% performance-per-dollar improvement on a serving tier is the kind of delta that filters into rental pricing with a lag — the same dynamic I described when 1T-parameter serving hit 1000 tok/s on B200s. Cheaper tokens on one cloud put downward pressure on the others.
The memory configuration also tells you where Google thinks inference is bottlenecked. 288 GB of HBM plus 384 MB of SRAM sized specifically for reasoning-model KV caches is an argument that long-context agent workloads fail on memory capacity, not on FLOPs. That is consistent with the NVHBM bandwidth argument and with every local-inference build I have done: the model that fits runs fast, the model that spills falls off a cliff.
Where This Falls Short
There are no independent benchmarks yet. No third-party tok/s numbers, no p95 latency figures, no measured performance-per-dollar. Google's 80% and 97% goodput figures are vendor claims from a launch post, and I would not plan a migration around them.
Both chips require Google's fourth-generation liquid cooling technology, which rules out anything resembling a conventional deployment. Access is via Google Cloud's AI Hypercomputer stack with bare metal options — framework support covers JAX, MaxText, PyTorch, SGLang, and vLLM.
Availability is "later this year" with no date and no published pricing. Requesting information is the only action available today.
For a homelab or a single workstation, nothing here changes your build. For anyone paying per-token on a serving tier, the 8i performance-per-dollar claim is worth tracking — but verify it against published pricing when it lands, not against the launch blog.
Sources: