Stop Measuring Agent Infrastructure by Gateway Latency Alone (opens in new tab)

I've been watching the LLM gateway benchmarks get faster. Bifrost at 11 microseconds, Helicone at 8 milliseconds, LiteLLM at 8ms. On single requests, the math is brutal: Bifrost is 720x faster than LiteLLM. But this week I watched three teams benchmark gateways, pick based on latency, deploy to production, and then realize they'd solved the wrong problem. The issue isn't the benchmarks. The issue is what they're benchmarking. And what they're not. The Latency Benchmark Measures a Chat Interfa...