AWS adds live KV cache-aware request routing to SageMaker AI
KV cache-aware routing uses live cache state to send long or repeated prompts to the GPU instance holding the longest match, falling back to load-aware routing when no match is available.
2 min readRead story



