What was released?

Google Cloud has published TPU Sync, an open-source C++ library for moving KV caches and model weights on Cloud TPUs. It is licensed under Apache-2.0, with Python wheels for PyTorch and JAX.

A KV cache stores a language model's earlier computations so it can generate responses faster. TPU Sync treats TPU high-bandwidth memory (HBM), host DRAM, shared storage and the data center network as a single memory hierarchy.

Why it matters

According to Google, the bottleneck is no longer computing model state but moving it. In internal tests run in July 2026, KV transfer latency fell by more than 90% on 235B to 480B parameter MoE models; on Qwen3 235B, transfer time dropped from several seconds to roughly 230 milliseconds. Google also reports up to 90% lower time to first token.

Technical details

  • Data moves via direct DMA in its native TPU tile layout, without copying.
  • Transfers run in the background while computation continues.
  • There is no XLA compilation in the data path; transfers and resharding run in native C++.
  • Google reports up to 223 GB/s PCIe bandwidth per Ironwood (TPU v7x) host and roughly 192 Gbps of network streaming on a single NIC.

Limitations

All figures are self-reported and Google says real-world results will vary. No reinforcement learning numbers were published, and the README states the library is not yet recommended for general use.