Pop quiz: at what point in the context length of a coding agent are cached reads costing you half of the next API call? By 50,000 tokens, your conversation’s costs are probably being dominated by cache reads.

Let’s take a step back. We’ve previously written about how coding agents work: they post the conversation thus far to the LLM, and continue doing that in a loop as long as the LLM is requesting tool calls. When there are no more tools to run, the loop waits for user input, and the whole cycle starts over. Visually:

Or, in code form:

def loop(llm):
msg = user_input()
while True:
output, tool_calls = llm(msg)
print("Agent: ", output)
if tool_calls:
msg = [handle_tool_call(tc)
for tc in tool_calls]
else:
msg = user_input()

The LLM providers charge for input tokens, cache writes, output tokens, and cache reads. It’s a little tricky: you indicate in your prompt to cache up to a certain point (usually the end), and you get charged as “cache write” and not input. The previous turn’s output becomes the next turn’s cache write. Visually:

Here, the colors and numbers indicate the costs making up the nth call to the LLM. Every subsequent call reads the story so far from the cache, writes the previous call’s output to the cache (as well as any new input), and gets an output. The area represents the cost, though in this diagram, it’s not quite drawn to scale. Add up all the rectangles, and that’s the total cost.

That triangle emerging for cache reads? That’s the scary quadratic!

How scary is the quadratic? Pretty squarey! I took a rather ho-hum feature implementation conversation, and visualized it like the diagram above. The area corresponds to cost: the width of every rectangle is the number of tokens and the height is the cost per token. As the conversation evolves, more and more of the cost is the long thin lines across the bottom that correspond to cache reads.

The whole conversation cost $12.93 total or so. You can see that as the conversation continues, the cache reads dominate. At the end of the conversation, cache reads are 87% of the total cost. They were half the cost at 27,500 tokens!

This conversation is just one example. Does this happen generally? exe.dev’s LLM gateway keeps track of the costs we’re incurring. We do not store the messages themselves going past, but we do keep track of the number of tokens. The following graph shows the "cumulative cost" visualization for many Shelley conversations, not just my own. I sampled 250 conversations from the data randomly.

The x-axis is the context length, and the y-axis is the cumulative cost up to that point. The left graph is all the costs and the right graph is just the cache reads. You can mouse over to find a given conversation on both graphs. The box plots below show the distribution of input tokens and output tokens.

The cost curves are all different because every conversation is different. Some conversations write a lot of code, so spend more money on expensive output tokens. Some conversations read lots of the code base, so spend money on tool call outputs, which look like cache writes. Some conversations waited for the user while the cache expired, so have to re-write data to the cache. In our data, the median input was about 285 tokens and the median output was about 100, but the distribution is pretty wide.

Let’s look at how some conversations got to 100,000 tokens. We sampled from the same data set, but excluded short conversations under 20 calls and also excluded conversations that didn’t get to 100,000 tokens. The number of LLM calls in the conversation matters quite a bit. The cache read cost isn’t really the number of tokens squared; it’s the number of tokens times the number of calls, and different conversations have very different numbers of LLM calls!

To go back to our original question, we can build a little simulator. Anthropic’s rates are x for input, 1.25x for cache write, 5x for output, and x/10 for cache read, where x = $5 per million tokens for Opus 4.5. In the default settings of the simulator, it only takes 20,000 tokens to get to the point where cache reads dominate.

As a coding agent developer and as an agent loop user, this cost structure gives me a lot to think about!

One metaphor for this is "dead reckoning." If we let the agent navigate a long task without feedback (in the form of tool calls and lots of back and forth), it will be cheaper, but, on the other hand, we know that it’s the feedback that lets the agent find the right destination. All things being equal, fewer LLM calls is cheaper, but is the agent’s internal compass off and is it going off in the wrong direction?

Some coding agents (Shelley included!) refuse to return a large tool output back to the agent after some threshold. This is a mistake: it’s going to read the whole file, and it may as well do it in one call rather than five.

Subagents and tools that themselves call out to LLMs are a way of doing iteration outside of the main context window. Shelley uses a "keyword search" tool, for example, as an LLM-assisted grep.

Starting conversations over might feel like it loses too much context, but the tokens spent on re-establishing context are very likely cheaper than the tokens spent on continuing the conversation, and often the effect will be the same. It always feels wasteful to start a new conversation, but then I remember that I start new conversations from my git repo all the time when I start on a new task; why is continuing an existing task any different?

Are cost management, context management, and agent orchestrations all really the same problem? Is work like Recursive Language Models the right approach?

These issues are very much on our minds as we work on exe.dev and Shelley. Let us know what you think!