Google Turned LLM Load Balancing Into Scheduling. What That Means for the Rest of Us (opens in new tab)

For workloads that send the same large prompt over and over, where a request runs can decide whether the model reuses expensive work or pays for it again\. Picture two LLM requests that arrive a few seconds apart\. Both carry the same 2,000 token block of context: a policy, an output schema, a ranking rubric, and a few examples\. The only thing that differs is the last 50 tokens, where each request includes a different customer query\. The first request lands on one replica\. The model reads ...