What is llm-d 1.0?
llm-d, launched in 2025 by Google Cloud, Red Hat, IBM, CoreWeave and NVIDIA, has reached its first stable release. The open-source, Kubernetes-native stack turns vLLM or SGLang model servers into one coordinated pool. The project is Apache 2.0 licensed and a CNCF Sandbox project.
What it solves
Round-robin load balancing breaks down for agent and reinforcement learning workloads. llm-d 1.0 routes each request to the replica that already holds its cached context, using prefix-aware routing, KV cache offloading and prefill/decode disaggregation to cut latency.
What's new in 1.0
- Multi-tenant quality of service with priority and fairness-aware flow control
- Stable v1 APIs and a hardened request path
- Zero-downtime upgrades with router high availability and DisaggregatedSet
- Agent-aware serving: session-aware routing and native Responses API state
- Async inference pulling jobs from Pub/Sub or Redis
- Expanded TPU support via TorchTPU
Performance
According to vendor-reported benchmarks on a 17,000-node multi-region GKE deployment, median time to first token fell by more than 85% and throughput rose 2.4x on an Ironwood TPU coding workload. For RL time-slicing, accelerator duty cycle rises only to just above 70%, leaving roughly 30% idle time.
