llm-d 1.0 brings distributed LLM inference to production operations

What is changing?

The llm-d community released version 1.0 of its open-source stack for distributed large-language-model inference on Kubernetes. The project is designed to run vLLM and SGLang servers, NVIDIA, AMD, and Intel GPUs, and Google TPUs behind a shared routing and scheduling layer.

The release adds priority- and fairness-aware flow control for multi-tenant workloads, a highly available router, zero-downtime upgrades, and a stable v1 configuration API. For agent workloads, session-aware routing can send follow-up requests to the backend that already holds the relevant KV cache, rebalancing when that replica becomes saturated.

llm-d is a CNCF Sandbox project developed by more than 500 contributors across over 25 organizations. For production teams, the main change is operational: upgrading, observing, and managing mixed hardware fleets with different traffic priorities.