In our previous warehouse we had roughly a year of headroom left. After moving to ClickHouse we have three to five years on current growth projections, steady-state cost is down about 70%, and usage data reaches customers around half an hour sooner.
Moving the data was straightforward. Proving that the new database produced correct billing results was the difficult part.
These two tables decide what customers get charged. A wrong number in them isn't an error a client retries and forgets: it becomes a line item on an invoice, it erodes a customer's confidence in everything else we report, and it lands as work on our support and finance teams a month after the fact.
So rather than cut over and watch for problems, we ran both databases in parallel for six weeks and shadowed one against the other — every query answered twice, the customer's answer coming from the existing store and the new store's answer going to a background comparison. Most of the engineering effort went into a question we hadn't anticipated: deciding which of the two systems' disagreements meant something was broken, and which meant everything was working.
This post covers the pipeline, why we moved, how we validated the new store, and what we got for it.
How the pipeline works#
Customer workloads run on Temporal Server cells, the isolated regional deployments that make up Temporal Cloud. Everything downstream of them exists to turn what those cells observe into a number on an invoice.
Figure 1. The metering pipeline. The highlighted repository is the only component that changed databases. The two coloured arrows are the read paths we re-implemented against it: violet, the customer-facing usage and billing APIs, migrated first; orange, the scheduled aggregation workflow, migrated last. Everything else is unchanged.
Following the diagram:
- As workloads run, each cell writes raw usage data to object storage.
- A worker reads that raw data, aggregates it into hourly totals, and publishes it onto a metering stream.
- An ingestion stage transforms and loads those aggregates into the meteringevents table.
- A scheduled workflow in our control plane reads deduplicated metering data for an hour that has settled and converts it into billable quantities.
- It sends those quantities to our Billing Platform, which produces invoices, and publishes a corresponding record onto a billing stream.
- A second ingestion stage loads those records into the
billingeventstable. - The customer-facing usage and billing APIs read from there.
One property of this pipeline shaped the migration more than any other: both streams guarantee at-least-once delivery. That's what makes them reliable, and it means both tables contain duplicate events by design. Every read has to collapse those duplicates before the number means anything.
We migrated the read path and nothing else: the few dozen queries that turn those two tables into numbers. Those are the two coloured arrows in the diagram, the customer-facing APIs reading billing events and the scheduled workflow reading metering events. The pipeline shape didn't change, and neither did the streams. What changed is which database answers those queries.
Why we moved#
Collapsing those duplicates is what pushed us to move. In our previous warehouse it happened in a scheduled batch job, and that job had two problems.
It was resource-intensive, so it competed with everything else running on the same cluster. And it sat directly between a customer using the product and that customer being able to see their usage, which meant that when it fell behind, usage data went stale and there was nothing to do but wait.
The more pressing issue was headroom. Extrapolating our growth against what that job could keep up with, we had somewhere around a year before the pattern stopped working. That's enough time to plan a migration and not much more.
ClickHouse handles deduplication differently: duplicate rows are collapsed at read time, as part of the query, rather than by a job that has to finish first. That removed the scheduled dependency from the critical path, and with it a category of failure where the data is correct but late.
Why code review wasn't enough#
Re-implementing those queries meant a different engine, a different SQL dialect, different deduplication semantics, and different behavior when numbers get large.
Code review alone couldn't establish that the new queries were semantically equivalent at production scale. A ported query can be idiomatic, compile cleanly, pass its unit tests, and still return a well-formed number that's wrong, because test fixtures are small and the interesting failures only appear at real volumes. Reading the diff tells you the query looks reasonable, not that it computes the same thing.
Hence the shadowing. The basic technique is well established — run the old and new systems side by side and diff the results. What took more work than we expected was designing a comparison that told us anything useful.
Deciding which differences were bugs#
Two systems reading the same data don't agree perfectly, and most of the reasons are uninteresting. Before the comparison could be useful, we had to write down which differences didn't count.
We settled on three categories.
Tolerated. Sums of floating-point values are compared within a relative tolerance rather than exactly. Floating-point addition isn't associative, so two engines that divide the work differently will add identical rows in a different order and reach marginally different totals. Neither result is wrong. Row counts, by contrast, were expected to match exactly.
Expected. Some differences were the goal of the migration rather than a threat to it, and the clearest one took us a while to accept. Queries for recent usage consistently returned more data from ClickHouse than from the old store. We went looking for duplicate writes and aggregation bugs before concluding that nothing was broken: the additional rows reflected fresher data, because reads were no longer waiting on a batch job. So we adjusted the comparison rather than the pipeline, moving it to time windows old enough that both systems had settled, and reclassifying "the new store is ahead" as a difference to log and ignore.
Figure 2. The freshness gap between the two stores, holding steady across the window — roughly half an hour on billing data and twenty minutes on metering data. Queries bounded to recent hours ran straight into it, which registered as a test failure until we reclassified it.
Blocking. Everything else: a value mismatch traceable to logic, or a value that couldn't be correct on its face.
That last category is where the comparison earned its keep.
The defect this caught#
A couple of days into shadowing, the comparison began reporting negative usage totals from ClickHouse. Because usage totals can't be negative, the sign pointed immediately at an overflow — we didn't need the other system's answer to know this one was wrong.
The column holding metering values was a 64-bit integer, and ClickHouse sums integer columns into an integer accumulator rather than widening it automatically. Storage totals for our largest accounts, aggregated over a long enough window, exceeded what that accumulator holds and wrapped into negative numbers.
Each individual decision looked reasonable in isolation. The column type was sensible. The query was idiomatic. It compiled and its tests passed, because test fixtures don't contain numbers that large. What identified it was running the query against production volumes and checking whether the answer was possible at all.
The defect was caught well before that query ever served a customer request. Shadowing was running months ahead of any cutover, so the old store answered every request during that period and the negative value existed only in a comparison log. We fixed it by casting before aggregating, added a test asserting that every billing query does so, and wrote the rule down where the next person working in that code would find it.
One rollout decision proved especially valuable here, though we hadn't made it for this reason. An integer overflow only appears once totals are large enough, so it's effectively invisible on a small account — and we had enabled shadowing first on a handful of high-volume internal accounts. Had we staged it the intuitive way, smallest and safest accounts first, the two systems would have agreed perfectly and we would have concluded the port was correct.
Comparison coverage and cutover therefore want opposite orderings. Comparison is most valuable on the accounts whose scale and usage patterns exercise unusual cases, so point it at the largest and strangest first. Cutover exposes you in proportion to blast radius, so start that at the other end. A comparison that has never surprised you is weak evidence that your systems agree.
Cutting over in order of reversibility#
Data moves through our pipeline in one direction: usage is counted, it lands in the metering table, a workflow aggregates it, billable quantities go to the Billing Platform, and the APIs read the result.
We cut over in the opposite order, taking the customer-facing read APIs first (violet in Figure 1) and the billing aggregation last (orange).
The reasoning was how expensive each mistake would be to undo. The read APIs are per-account and read-only, so a wrong answer affects one account's view and is reversed by changing a flag. The aggregation path is different in kind: its output goes to the Billing Platform, and by the time a wrong number is noticed, it has already left the system that produced it.
So the paths that were cheap to reverse went first and served as the proving ground, running a full phase ahead of everything else. The path whose output we couldn't take back was the last thing we touched, by which point the queries feeding it had been compared against the old store for weeks.
Each path moved through the same three phases, one account at a time: shadow both stores but serve from the old one; shadow both and serve from the new one; stop reading the old one. Separating comparison from serving let us watch each account before changing anything customer-visible, and rolling back was a configuration change rather than a deployment.
Figure 3. The rollout, one account at a time. (a) Before: every read goes to the legacy store. (b) The reader queries both, compares them in the background, and still serves the legacy answer — nothing a customer can see has changed. (c) Same two reads and the same comparison, but the served answer now comes from the new store, which remains one flag away from rollback. (d) Only the new store is read. Note what leaves with the second read: the comparison goes too, which is why the previous phase is the one worth lingering in.
What it bought us#
Headroom. This was the point of the exercise. Where we had roughly a year of runway before, our growth projections now give us three to five years, and the practical ceiling is well beyond that: ClickHouse comfortably handles datasets several orders of magnitude larger than what we process today.
Cost. Our design doc set an aspirational target of a 30–50% reduction and treated it as a secondary criterion, since the operational case stood on its own. Steady-state spend came in around 70% lower. Little of that is cleverness on our part: ClickHouse Cloud keeps data on object storage, which prices differently enough that the saving is largely structural.
Query latency. Across a week of running both systems against identical live traffic, the same queries averaged about 1.4 seconds faster on ClickHouse. That's an average across query types rather than a percentile breakdown, and the week was mid-month, so it doesn't capture month-end billing load.
Freshness. Usage data reaches customers about half an hour sooner, because reads no longer wait on a batch job to finish. This is the improvement that first showed up as a test failure.
Operational load. The scheduled deduplication job is gone, along with the metrics and alerts that existed to tell us when it was falling behind. Those weren't re-pointed at the new store; there was no longer anything to point them at.
Schema changes. Some weeks after the cutover we needed to widen the type on a metering value column, on a table holding billions of rows. It ran as a single statement against live traffic and completed in about five minutes, with no maintenance window and no coordination. On the old pattern, a change like that meant reasoning about refresh behavior and scheduling around it.
What we'd do again#
Write down what "the same answer" means before you start comparing. Tolerances, and the differences that don't count, are part of the design. Without them a comparison either drowns you in noise or quietly certifies a real defect as normal.
Check whether an answer is possible, not only whether it matches. Our most useful signal wasn't equality between two systems. It was noticing that one number couldn't be correct regardless of what the other one said.
Order the rollout by cost of reversal. That's often the opposite direction from the way data flows through the system, and it means the component you're most worried about is the last one you change rather than the first.