700 tokens per second: The new speed benchmark for Google's AI chips

For businesses relying on artificial intelligence, the options for processing power just expanded. Meta, the company behind the popular PyTorch framework, is now using Google’s custom AI chips in live production. This move marks the first time Google has sold its Tensor Processing Units, or TPUs, to an external customer. It signals a significant shift in how major tech companies handle their AI workloads. The change means that the hardware landscape is no longer just about Nvidia.

The transition was made possible by a new version of the SGLang inference engine. Previously, running these chips required using Google’s internal JAX framework. The new software allows PyTorch to run natively on TPUs without major code rewrites. A matching open-source version of the vLLM engine was also released. This lowers the barrier for other developers who want to use this hardware for their own AI projects.

Zongwei Zhou, the PyTorch-TPU lead at Meta, confirmed the deployment. He stated that “SGLang-TorchTPU is running in production at Meta today.” This followed a close collaboration between Meta, Google, and the SGLang team. The work was done by RadixArk, a startup founded by the creators of SGLang. Zhou described the result as a “remarkably mature, PyTorch-native serving stack.”

Meta has long preferred to keep its software options open. Jana van Greunen, director of PyTorch engineering at Meta, explained that the company “cannot be constrained by a single hardware architecture.” The new TorchTPU software “enables us to deploy our PyTorch workloads to Google TPUs.” Google confirmed it co-developed this software with Meta’s team to ensure smooth integration.

This is not the first time Meta has touched Google’s hardware. Older tools like PyTorch/XLA served as a bridge for TPUs. Reports suggest Meta research teams used these chips from 2020 to 2023 before canceling those contracts. The current push is different because it offers a native solution rather than a workaround. Google has focused heavily on Meta because the company did not want to migrate its entire stack to JAX.

The physical location of these chips remains a detail of the agreement. Talks reportedly included rentals through Google Cloud in 2026, with potential purchases in 2027. Meta later signed a multi-year rental agreement. The exact servers are likely in Google data centers, but the production status is confirmed.

The timing of this announcement is notable. It came one day after Amazon open-sourced its own PyTorch backend for its Trainium chips. Both moves aim to reduce the software lock-in provided by Nvidia. By making alternative chips easier to use with standard tools, these companies are trying to break the monopoly on AI development environments.

Performance numbers back up the utility of the new setup. A tuned vLLM configuration served a massive 2.8-trillion-parameter model at over 700 tokens per second. This speed was achieved on seventh-generation Ironwood TPUs. The test used 16 chips with speculative decoding. In comparison, Nvidia’s GB200 chips reached 452 tokens per second in a similar setup. At small batch sizes, the new chips delivered roughly 1.4 to 2 times the throughput of the Nvidia baseline.

The software ecosystem is expected to grow quickly. SGLang-TorchTPU will add support for new models like DeepSeek V4.1 Flash. RadixArk aims to support new open models on their release day by the first quarter of 2027. Google, Meta, and RadixArk will share more details at the SGLang Summit in San Francisco in November. For now, the market has a viable, fast alternative to the traditional leader.