We Measured LLM Prompt Caching in Production — Same Prompt, 0% to 91% Hit Rates (opens in new tab)

We run an AI companion bot. Every chat turn, the model sees the same ~5K-token prefix — character persona, content-tier rules, formatting guardrails, a memory blob — plus one new user line. Without caching, we pay for those 5K input tokens every single turn. So we turned on prompt caching across the providers we route through, measured it, and the spread was bigger than any of the marketing pages prepared us for. Here's the table that survived four weeks in production, plus the one gotcha tha...