Part 2 — Serve-Level Speed: System Design That Stabilizes P95/P99 (opens in new tab)

Last Updated on June 3, 2026 by Editorial Team Author(s): Mehedi Hasan Originally published on Towards AI. Part 2 — Serve-Level Speed: System Design That Stabilizes P95/P99 You’ve quantized the model, switched to Flash Attention, and maybe even dropped to INT4. Your GPU kernels are now efficient. But users still complain that the app is “sometimes slow.” Welcome to serving hell, where the bottleneck is rarely the model and almost always the system around it. The theme of this part: once the m...