Google announces native PyTorch TPU backends for vLLM and SGLang
The TorchTPU engines add a PyTorch serving path alongside existing JAX runtimes. Project documentation limits what the launch establishes about model coverage and deployment.
By Benchdict AI DeskPublished Oct 21, 20264 min read
Google announced native PyTorch TPU backends for vLLM and SGLang on October 20, developed with Inferact and RadixArk, respectively. The launch adds a TorchTPU serving path rather than introducing TPU support for the first time.
Key facts
- New runtime
- Native PyTorch TPU serving through TorchTPU
- vLLM deployment
- Documentation directs PyTorch-native TPU users to the separately released vllm-torchtpu project
- SGLang deployment
- OpenAI-compatible serving API; development and testing focused on Ironwood
- Existing deployments
- Google says the JAX runtimes will remain maintained and supported
What changes beneath the serving API
Google says the new engines retain their frameworks’ worker models, schedulers, KV cache managers and API servers while adding TPU implementations underneath. TorchTPU uses standard PyTorch tensors, supports eager execution or XLA compilation through torch.compile, and routes collectives through torch.distributed. Pallas kernels run as custom operators inside the compiled graph.
That separates two aspects of the launch: familiar serving interfaces remain, but the execution path changes. For developers customizing or debugging inference in PyTorch, the addition is a native runtime option—not a requirement to abandon an existing JAX deployment.
The deployment documentation makes that distinction concrete:
| Area | Documented native-PyTorch path |
|---|---|
| vLLM project | The vLLM documentation directs users to vllm-torchtpu, which has separate releases. |
| SGLang setup | SGLang directs users to run TPU serving recipes from the TorchTPU checkout with a matching runtime. |
| SGLang interface | The backend supports SGLang’s OpenAI-compatible serving API. |
| SGLang coverage | Available feature combinations depend on the model and topology; development and testing focus on Ironwood. |
Prefix reuse is supported; universal model parity is not established
For multi-turn agent workloads, Google says both engines can retain reusable prefixes in high-bandwidth memory and automatically move older prefixes into host memory when host offload is enabled. Both use the open-source C++ TPU Sync library for KV cache transfer and management, according to Google.
Google says both engines are running at production scale with preview customers and that SGLang-TorchTPU is running in production at Meta. Those are Google’s deployment claims, not independently confirmed customer accounts.
Model-specific documentation gives a narrower picture than the broad production language. SGLang’s documented Kimi-K3 coverage includes a small-model compile test using dummy weights. That does not establish full-size accuracy validation or every serving-feature combination.
It also does not invalidate Google’s separate vLLM performance claim: the company reports more than 700 tokens per second for one user serving Kimi K3 with an integrated Pallas megakernel on Ironwood. The engine and validation scope matter; a result for one is not evidence of equivalent coverage in the other.
An additional TPU path, not a GPU cost comparison
The October release follows the July RadixArk–Google partnership announcement, which promised same-day TPU support for new open models when they became available on GPU. Google now describes day-zero model support as an area of investment, rather than establishing universal coverage.
Google also reports a lowest cost of 2.01 cents per million tokens after optimizing its Qwen 3.5 submission on Ironwood. That is a reported benchmark result, not a published service tariff. Without the configuration and cost methodology, it does not establish a comparable GPU deployment cost.
The immediate change for serving teams is therefore specific: a native PyTorch option now sits alongside the supported JAX runtimes, while deployment details and demonstrated coverage remain project- and model-dependent.
Not yet confirmed
- The October launch and its performance claims lack confirmation from two independent outlets in the available evidence. Meta’s reported deployment is attributed to Google.
- The benchmark configuration, token mix, concurrency, latency targets, hardware allocation and accounting behind the 2.01-cent result are not established.
- Formal general-availability status, service-level commitments and regional availability are not established.
- Exact compatible native-PyTorch release versions and installation requirements are not established; SGLang specifies a matching TorchTPU runtime.
- Full-size accuracy validation and per-model feature parity across both native engines are not established.
What to watch
- Publication of compatible TorchTPU and serving-engine release matrices.
- Expanded model-specific validation and topology coverage in SGLang documentation.
- Reproducible benchmark configurations and independently confirmed production deployments.
Sources
- Announcing New vLLM and SGLang Inference Engines on TPUGoogle Developers Blog, Oct 20, 2026.
- vLLM JAXdocs.vllm.ai, Oct 21, 2026.
- TPU - SGLang DocumentationSGLang Documentation, Oct 21, 2026.
- RadixArk Joins Forces with Google to Bring Full SGLang Features to TPUslmsys.org, Oct 21, 2026.
- Google Cloud Backs SGLang to Kill the GPU-to-TPU Migration Tax | AlphaSignalalphasignal.ai, Jul 30, 2026.
Benchdict AI Desk, AI-assisted reporting desk. Stories follow our editorial standards; corrections are logged above and on the corrections page.