PyTorch-based inference on TPU

Google announced that vLLM and SGLang inference engines now run on TPU with native PyTorch backends. Powered by TorchTPU, the integration lets developers run and optimize serving workloads using familiar PyTorch tools.

What's included

  • TPU-optimized kernel implementations
  • KV cache management with TPU Sync
  • Disaggregated serving support

These features target teams aiming for low latency and high throughput in production.

Who it's for

The engines were built in collaboration with open-source project maintainers. Developers in the PyTorch ecosystem can move existing models to TPU without an extra proprietary software layer.

Performance claims

Google's public benchmarks demonstrate state-of-the-art performance and industry-leading cost efficiency on Ironwood TPU across long-context and agentic workloads. The company says the engines are production-ready.