In the last post we walked through why AI tools that feel sharp in the first few minutes can feel noticeably duller an hour later, and we called that context rot. That post was about quality. This one is about the other thing that happens as a session grows: cost. It is easy to assume these are the same problem wearing two hats. They are not. They come from two different mechanics, and a team can manage one perfectly while staying completely exposed on the other.
The Reuse Mechanic
Every AI conversation is stateless. Nothing persists in the system's head between your messages the way it would for a person. Each time you send a new message, the entire conversation so far gets handed back to the model in full and read again before it writes a reply. We used the case-file metaphor for this last time, and it still holds.
What we did not cover is what it costs to keep rereading that file. The expensive part of reading a file is not the reading itself, it is the work the system does to understand what it is reading. That work only has to happen once per piece of information. If the earlier parts of a conversation have not changed, the system can reuse what it already worked out instead of redoing it from scratch. That reuse is commonly called caching, and it is why a long, stable conversation stays cheap even as it grows.
The catch is that this only works if the earlier material stays exactly the same. Change something upstream, switch which model you are using mid-conversation, adjust a permission, resume a session after a tool update, and the system can no longer trust its earlier work. It has to redo it, at full cost, before it can pick the cheap path back up.
Cache cost simulator
Drag the slider to add turns to a session. Toggle the checkbox to see what one mid-session change, like switching models, does to relative cost.
Figures are relative cost units scaled from published cache write and read multipliers, not a dollar estimate. They are meant to show the shape of the curve, not a specific bill.
Two Axes, Not One
Quality and cost degrade for different reasons, on different timelines, and they need different habits to manage.
| Context rot | Cache rebuilding | |
|---|---|---|
| What breaks | Output quality, drift, weaker recall | Your bill, unexpected cost spikes |
| What causes it | Too much accumulated, stale, or irrelevant material in the conversation | Something upstream changed, forcing the system to redo work it had already done |
| How it shows up | The AI starts contradicting earlier instructions or losing track of details | A session that was running smoothly suddenly costs more, with no obvious reason why |
| How you manage it | Keep sessions narrow, hand off with a summary, do not let one thread sprawl | Keep the foundation of a session stable, batch changes at natural breakpoints |
A team can be excellent at one of these and still exposed on the other. Tightly scoped conversations that never suffer from drift can still be paying full price on every message because someone keeps switching tools or settings mid-session. A perfectly stable setup can still produce degraded answers if nobody is managing how much irrelevant material piles up. Both disciplines matter, and neither substitutes for the other.
Where This Bites Hardest
For a simple chatbot used for occasional questions, this mostly stays theoretical. It becomes a real operational concern with agentic tools, the kind that do not just answer questions but actively do work: reading files, running commands, executing multi-step tasks over long sessions. Claude Code is a good example, and we touched on it briefly last time for how quickly it consumes context. The cost side compounds that.
These sessions run longer by nature and accumulate more, project files, command output, results from earlier steps. That makes the foundation bigger, which raises the stakes when something forces a rebuild. A single mid-session change, switching the model to try something faster, adjusting an access permission, resuming after a tool update, can force the entire accumulated foundation to be reprocessed from scratch. On a long-running task, that one change can be the most expensive moment of the whole session, and it rarely comes with an explanation attached. It just shows up as a session that cost noticeably more than the one before it.
How to Manage Both
Treat these as two checklists, not one. Good habits tend to reinforce both, but they are worth naming separately so nothing gets missed.
For quality
- Keep sessions scoped to a single task, and start fresh when the task changes.
- Front-load what actually matters, so the important context stays salient.
- Hand off deliberately. When a session runs long, summarize what matters and start clean rather than dragging the entire history forward. A useful approach is to have the chatbot give you a handoff file that you can upload as context in a fresh session.
For cost
- Keep the foundation stable. Avoid switching models, permissions, or settings mid-session if you can help it. If something needs to change, do it at a natural break, not mid-task.
- Batch changes together at session boundaries rather than one at a time mid-stream, each change can trigger its own rebuild.
- Watch for the cold-start moment. Resuming a session after a long idle period or a tool update is often the single most expensive request you will send.
- Match the session to the task. A short, fresh request is often both cheaper and more reliable than reviving a long, stale one.
The Core Idea
Context rot and cache rebuilding are not the same problem wearing two hats. They are two separate mechanics, one governing what you get back and one governing what you pay for it, and treating them separately is what lets you actually manage them instead of reacting to symptoms after the fact. That is the same thesis behind this whole series: the biggest returns usually come from understanding the unglamorous mechanics well enough to make good decisions about them, not from chasing whatever feature shipped most recently.
If the last post gave you a way to think about why long sessions get less reliable, this one is the other half, why they can also get quietly more expensive, and what to do about both at the same time.
Part of the Proof of Value series from Busted Eye. Submit your details below if you want to talk through what this means for your team.