PyTorch goes native on TPUs as the post-CUDA stack ships

The CUDA-only era of serving ends

Google announced native PyTorch backends for vLLM and SGLang on October 20, powered by TorchTPU — two of the open-source serving engines now run on TPUs without a JAX rewrite. The backends ship with production features: KV cache management with TPU Sync, optimized TPU kernel implementations, and disaggregated serving, where prefill and decode are split across different hardware for better throughput. m:…7010

Google says public benchmarks demonstrate state-of-the-art performance and industry-leading cost efficiency on Ironwood TPU. Those are Google's own benchmarks, so take the superlatives with the appropriate discount — but the target workloads are telling: Google says the performance and efficiency gains apply to long-context and agentic workloads, which is exactly the shape of work agents generate, long-running sessions with heavy KV caches. m:…7010

Ray: migration as find-and-replace

Anyscale and Google Cloud announced native Ray support for PyTorch on TPUs the same day. The pitch to developers is deliberately boring: migrating existing PyTorch code from GPUs to TPUs involves changing 'cuda' to 'tpu' and switching the distributed communications backend from 'nccl' to 'tpu_dist'. Ray runs one worker per TPU chip, matching the single-device-per-worker model GPU developers already use, and isolates individual TPU devices to workers using TPU_VISIBLE_CHIPS. m:…8860

The plumbing underneath is aimed at production teams: TorchTrainer automatically provisions a SlicePlacementGroup when configured for TPUs, Ray Train automates PyTorch's distributed environment variables on TPUs, vLLM-TorchTPU registers TorchTPU as a vLLM platform plugin, and Ray Serve LLM deploys OpenAI-compatible vLLM endpoints across TPU slices. Anyscale also announced a Special Interest Group for Ray on hardware accelerators with a vendor-neutral roadmap — Ray already supports more than a dozen kinds of hardware accelerators, and the SIG is meant to keep that support honest across vendors. m:…8860 m:…1033

llm-d 1.0 productionizes agent serving

The llm-d project released version 1.0 on October 20, an open-source framework for production-scale LLM inference, with more than 500 contributors from over 25 organizations. The 1.0 feature set reads like an answer to the agent era: multi-tenant quality of service with priority and fairness-aware flow control, and agent-aware serving through session-aware routing and turn-priority scheduling — bursty agent workloads sharing a fleet without starving one another. m:…0296

Adoption is already real. Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine, DigitalOcean uses it as the foundation for its Serverless Inference and Managed Agents services, and JetBrains runs it in production serving traffic for its Junie Lite plan. IBM's own case study says IBM Research achieved 11.5 times lower inference cost with llm-d versus an all-proprietary configuration — IBM's own number, from IBM's own blog, so treat it as the company's best-case result rather than a third-party benchmark. IBM also says an llm-d deployment of GLM 5.2 delivered over 6.6 million output tokens per minute at peak — again, IBM's own reported result, not an independent measurement. m:…0296

Japan bets on photonics

Further out, Preferred Networks and Tenstorrent were selected by Japan's NEDO for a five-year program on xPU design incorporating photonics-electronics integration. PFN will develop AI chiplets with ultra-high memory bandwidth using 3D memory stacking, while Tenstorrent will build CPU chiplets on an open architecture supporting confidential computing. The partners plan to commercialize the research outcomes for applications including AI agents, and the program sits inside NEDO's broader post-5G infrastructure research effort. The stated technical goal is system-level integration of AI processors for both training and inference. m:…7335