Context Windows Are Bigger and Dumber Than You Think (opens in new tab)

The Number You See Is Not What You Get When Anthropic announced Claude’s 200,000-token context window, or when Google unveiled Gemini 1.5 Pro with a million-token window, the coverage treated it as straightforward progress. More tokens in, more capability out. The framing makes intuitive sense: if a model can see more text at once, it should be able to reason about more text at once. This is not quite right. Context window size and context window effectiveness are two different things, and th...