The clearest gains came from cleanup rather than construction, and from writing things down where a future session would find them.
We recently moved the store behind Temporal Cloud's usage and billing data onto ClickHouse, and we used coding agents heavily throughout. For context behind that story, please read the previous blog. When the work was finished, we went back through the prompt history to understand what had actually helped.
The honest answer was not what we expected, and it is worth stating before anything else.
We mapped our saved prompts against the pull requests that made up the migration, and they covered most of the code we deleted and very little of the code we wrote. Some of that is a recording gap: our transcript history begins over a week into the project, so the sessions that produced the early scaffolding are simply missing. But the pattern held across everything we could see. The agents were most useful on the removal work, and the reason turned out to be less about the tools than about a few habits we had already adopted for other reasons.
Three of those habits did most of the work. None is specific to any particular agent or model, and all of them are cheap.
Why cleanup is the part that benefits#
Most migrations leave debris behind, and that debris tends to survive for years. Not because removing it is difficult, but because it is tedious and because nobody is confident about what will break.
A cleanup change touches a scattering of files that nothing connects except history: configuration entries, deployment manifests, environment variables whose consumers were never written down, test fixtures, mocks. Each individual edit is trivial. The hard part is establishing the complete list, and that is exactly the sort of exhaustive, boring search that coding agents can handle well and engineers often do reluctantly.
Our final teardown touched several dozen files, most of them deployment configuration nobody had looked at in years. It took a single working session. That is the change most teams defer indefinitely, and it is worth understanding why it became affordable, because the reason has very little to do with the model.
Practice 1: Write the cleanup instructions when you write the temporary code#
The scaffolding we built to validate the migration was always going to be thrown away. We knew that on the day we wrote it.
So we wrote that down, in the code, in a form that could be found later. Our codebase already had a convention for this: a distinctively named comment marker used for anything complex, surprising, or likely to matter to whoever came next. Every piece of temporary migration code got one, and each said what the code was for and what should happen to it when the migration finished.
Simplified, they looked like this:
// CLEANUP-NOTE: temporary comparison harness for the storage migration.
// Remove this package once the migration is complete; regenerate mocks afterward.
Several weeks later, those comments were the deletion checklist. An agent could find every one of them, read what each said, and act on it. In one case a note explicitly instructed that the package be removed in full, and that is what happened.
The important part is the timing. The instruction was written by the person who understood the debt, on the day the debt was created, when the reasoning was still fresh and the cost of recording it was close to zero. Nobody had to reconstruct months later what a piece of scaffolding was for. That reconstruction is usually the expensive step, and it is the step that gets skipped.
Practice 2: Put durable rules next to the code, and say how they are enforced#
The second habit came out of a production defect rather than a plan.
During the migration our comparison harness surfaced a bug in a ported query: aggregating a numeric column into a fixed-width integer accumulator overflowed and produced negative totals for our largest accounts. The immediate fix was small. What we did around the fix is the part worth describing.
Along with the correction, we added a regression test asserting that every affected query does the right thing, and a short instruction file in the same directory stating the rule in plain language. Simplified, that file read:
## Aggregating usage values
Usage values grow very large for high-volume accounts. Summing them into a
fixed-width integer column overflows into negative totals.
Rules:
- Always widen the type before aggregating these columns.
- The regression test in this package fails if any query builder emits a
bare sum over them.Two details made this more useful than a comment would have been.
The first came from review. The rule was originally written into the instruction file at the root of the repository, and a reviewer argued for moving it into the directory it applied to, on the grounds that agent instructions consume a limited context budget: everyone working anywhere in the codebase pays for a rule stored at the root, whether or not it is relevant to them. Scoping it to the package meant it loaded only for people and agents working in that area.
The second is the last line of the file. The rule names the test that enforces it. That sounds like a small thing, but it is what lets a future reader establish whether the rule is still real. A written rule that points at nothing is an assertion; a written rule that points at a failing test is checkable.
Practice 3: Treat agent instructions as artifacts that go stale#
Both practices above produce written instructions, which creates a maintenance obligation that is easy to overlook.
The instruction file from the second practice lived about a month. Once the underlying column type changed, the rule it described stopped being necessary, and a colleague deleted the file along with the now-redundant test as part of unrelated cleanup. That worked because the staleness was visible: the rule referred to a cast that was no longer in the code, so anyone reading the diff could see it had expired.
We did not manage this consistently. The same migration quietly invalidated a different instruction file elsewhere in the codebase, one that guides agents through adding a new kind of usage record. It still directs them to edit files that the migration deleted, and to update a data structure that has since become a function. Its verification steps tell the agent to build and test something that no longer exists.
Nothing caught this, and nothing was going to. These files are prose, and no compiler checks prose. If an agent follows those instructions today, it will fail, waste a cycle, and eventually work around them, and the only signal will be an engineer noticing the confusion.
So the third practice is really an admission. If you are going to leave instructions for future sessions, they belong in whatever process you use to keep documentation honest. Ours were not, and one of them is wrong right now.
What we cannot claim#
We would like to report that these practices made the migration dramatically faster. We cannot support that.
Our prompt record covers the cleanup far better than the construction, so any speed claim would rest on the half we can see least. We also have no comparison: we did not run the same migration twice, and we have no baseline for how long the removal work would have taken by hand. Our honest estimate is that most of that work would simply have stayed in the codebase for another year, which is a real benefit but not a measurable one.
The automation was thinner than it sounds, too. We ran unattended loops to handle continuous integration failures and review comments, and most of what they produced was hygiene that our own local tooling should have caught before the push. Nearly all of the agent instruction files these practices depended on also predated the migration by months. We did not build a toolkit for this work. We were the first heavy consumer of one that already existed.
There is also a limit that no amount of prompting removes. The most effective instructions we wrote were the ones carrying a carve-out: apply this change here, but not there, for a reason specific to how our billing pipeline is wired. That reason took years to learn, and no template supplies it. What writing the rule down changes is not whether you need to know it, but whether the agent can show you how it applied what you know, which a hand-written list of files cannot do.
The short version#
Write down what temporary code is for and what should happen to it, at the moment you create it. Keep durable rules next to the code they govern, and make each rule name the thing that enforces it. Then treat those written instructions like code that can rot, because nothing will tell you when it has.
We built a mechanism for one session to leave instructions for the next, used it to remove a system, and left instructions behind that still describe the system we removed.