The day before Replay, I walked into Target and bought every lazy Susan they had.
I was not having an unusually specific kitchen emergency. I needed carriers for 2,300 conference badges.
Each lazy Susan held two 16-port USB hubs. My coworkers could load 32 badges onto one, rotate it around to reach the cables, and hand the entire thing to a flashing station. The station would run Ignition, our Temporal-powered firmware flashing system, and hand the carrier to the next part of the assembly line.
This is the part of the badge project where the phrase "distributed systems" became extremely literal.
Flashing one ESP32 is a command. Flashing 2,300 of them, on six laptops, while more than 70 coworkers are physically assembling the hardware around you, is a production system. Devices disappear and return under new names. USB cables sit just loose enough to ruin your evening. A successful write does not prove that the device booted. A failure on badge 17 should not make an operator wonder what happened to the other 31.
I knew this problem was coming early. It was sitting in the back of my head when I started buying USB hubs before my trip to China. Espressif has factory tooling, and it is very good at putting bytes onto chips. I was worried about everything around that operation: retries I did not have to babysit, a record of what happened to each badge, tests that proved the new firmware actually worked, and a tool my coworkers could start without needing me beside them.
I thought Temporal could do that part better.
So we built Ignition.
The problem arrived by DHL#
The first version of Ignition existed weeks before the badges arrived. We tested it on development hardware and kept tightening the workflow as the firmware changed.
That testing had one fairly important blind spot.
The few badges I brought home from China had all been running production firmware for weeks. My test pile no longer contained a single example of the QA firmware running on the 2,300 badges still in transit. I knew QA firmware presented itself differently over USB. I had simply forgotten.
I was not sitting around waiting for DHL. The lanyards were the wrong size, so I bought every book ring I could find at Office Depot stores within 13 miles of San Francisco so the lanyards could actually connect to the badges. The venue had also forgotten that we needed a 2.4 GHz network, so I got that running too.
I was being productive. Just on every emergency except the USB behavior I had forgotten existed.
DHL delivered the production shipment on the evening of workshop day at Replay. The badges were due to be revealed during the closing keynote two mornings later. In practical terms, we had two nights and one full workday to assemble and flash all 2,300 of them. We had ordered 300 extras in case anything went wrong, but the extras also needed firmware. "Spare" does not mean much if the spare is still running a factory test image.
Then we plugged in the first real batch.
The QA firmware changed how the devices appeared to macOS when they reset into the ESP32 bootloader. A serial port could disappear and come back with a different name and identity. Ignition had detected a badge, but after the reset it could no longer be sure that the device now sitting at a new port was the same physical badge it had started with.
This was not theoretical hardware flakiness anymore. The devices on the table were the ones we had to give attendees.
Edward Amsden fixed the USB re-enumeration path. Ignition learned to recognize the QA firmware's USB serial pattern, pulse RTS and DTR to put the badges into their bootloaders, wait for the operating system to discover them again, and rebuild its view of the carrier using the post-reset ports. That recovery ran concurrently rather than turning into 32 little manual button rituals.
Most badges flashed normally after that. I do not want to retrofit a heroic failure rate into the story. USB re-enumeration was the issue we saw most often, and Ignition handled it. When something genuinely strange escaped the automation, I was already operating one of only six stations and could step in.
That was the goal. The boring path stayed boring, and the weird path left evidence.
A carrier became a Workflow boundary#
Ignition did not begin with the final batch model. An early version continuously watched for devices and started work as badges appeared. It sounded convenient when the unit of work was "whatever is currently connected to my laptop."
It felt much worse when the unit of work became a physical carrier moving through a room.
The lazy Susans gave us a better boundary. One carrier held exactly 32 badges. An operator plugged in both hubs, Ignition counted the devices, and the run refused to begin if it did not see the expected 32. That caught loose connections before a partially flashed carrier could travel farther down the line.
Once the count matched, the operator confirmed the batch. Ignition started one parent FlashBadgesWorkflow for the carrier and one BadgeFlashWorkflow child for every badge.
32-badge carrier
|
v
FlashBadgesWorkflow
|
+-- BadgeFlashWorkflow 01
+-- BadgeFlashWorkflow 02
+-- BadgeFlashWorkflow 03
| ...
+-- BadgeFlashWorkflow 32
| Two of our amazing Temporalites at our flashing station in the badge workroom (Mike, who will get his well-deserved kudos later in this blog, pictured on the left). Each carrier holds two 16-port hubs mapping directly to one parent Workflow with 32 per-badge child Workflows. |
This alignment between the room and the code turned out to matter. A parent Workflow was not an abstract grouping invented for software neatness. It represented the object someone physically handed to an operator. The Workflow completed when that carrier was ready to move again.
The explicit boundary also prevented an old run from becoming confused with the next hub swap. After a batch finished, we unplugged the complete carrier, sent it downstream, connected another one, and started a new Workflow. There was no watcher trying to decide whether 32 disappearing devices were failures or simply coworkers doing their jobs.
One physical badge, one durable state machine#
Inside a batch, every badge had its own child Workflow and its own state:
Resolve USB identity
|
v
Flash firmware + filesystem -- retry up to 3 times
|
v
Find the badge after reboot
|
v
Observe a known boot marker -- retry validation once
|
v
Synchronize and verify clock
|
v
PASS / FAIL
The Workflow owned coordination and policy. Activities performed the side effects: inspect USB devices, build artifacts, invoke esptool, read serial output, and set the clock. That separation is especially useful with hardware because an Activity can fail for reasons that have nothing to do with the logical process. A port may be busy. A cable may wobble. The operating system may take longer than usual to enumerate a device. The chip may reboot while the host still believes its old serial path exists.
Ignition resolved the current port before each flash attempt instead of assuming the first path would remain valid. The production Workflow allowed up to three full flash attempts, with a delay between them. Device detection had its own retry policy. Boot validation ran separately from flashing, so a successful esptool exit code could not quietly become our definition of a working badge.
That distinction saved us from a comforting but incomplete result. "The bytes were written" and "the badge is ready for an attendee" are not the same statement.
The flash Activity wrote the bootloader, partition table, OTA boot metadata, application firmware, and filesystem image in one esptool session at 921600 baud. Ignition found the application and filesystem offsets from the compiled partition table rather than relying on guesses tied to an environment name.
We learned that lesson the annoying way. An earlier fallback path used the wrong filesystem offset for the 16 MB production board and wrote into an OTA application slot. The fix parsed partitions.bin directly. Hardware is very good at converting a slightly wrong integer into an exciting afternoon.
While esptool ran, the Activity sent heartbeats with useful progress instead of stuffing an enormous serial transcript into Workflow state. Operators could see phases such as resolving USB, writing firmware and apps, verifying boot, and synchronizing the clock. Temporal history retained bounded log tails and final results for later inspection.
The useful question was no longer "Did the batch flash?" It was "Which badge failed, at which stage, after how many attempts, and what did it say before it stopped?"
Writing the image was only half the job#
After flashing, Ignition waited for the badge to return and opened its serial port at 115200 baud. The production firmware printed a known marker early in startup:
Temporal Replay 2026 - <build timestamp> - ...
Seeing that marker proved we had booted into the expected firmware far enough to initialize serial logging. The validator also watched for failure signatures including Guru Meditation errors, stack canaries, panics, aborts, backtraces, and brownout resets. A badge that wrote successfully and immediately crashed was a failed badge.
Boot verification received two attempts. After a valid boot, Ignition synchronized the badge's clock from the host and checked the reported epoch rather than assuming the command worked. BLE verification existed during some stages of development, but we deliberately skipped it in the final production path. I would rather describe the checks we actually ran than turn the Workflow into a fictional comprehensive hardware test suite after the fact.
There were still physical checks. Operators could look at all 32 screens and make sure the carrier had booted into the correct firmware. By then, though, Ignition had already resolved the recoverable errors and identified the failures it understood. The visual check was a final sanity pass, not our retry mechanism.
Six small factories instead of one important server#
At peak, we ran six flashing stations. Each one could process a 32-badge carrier, which gave us 192 badges in flight at once.
| That's me at one of our flashing stations. |
Every station was self-contained. A laptop ran its own local Temporal development server, its own set of 32 workers, and its own Ignition CLI. We did not create a shared Temporal deployment in the middle of the venue. I didn't need one.
This was partly architecture and partly survival. Our IT team brought a stack of spare computers specifically for the line. If a station became troublesome, we could replace the laptop instead of diagnosing it while carriers piled up behind the operator. The other five stations continued working because they did not share that failure domain.
The local setup also made training almost nonexistent. A coworker received a loaded carrier, connected it, started Ignition, and waited. Ignition verified that all 32 badges were present before starting, handled the retryable failures, and presented the final result. The coworker could glance across the screens before handing the carrier back.
This is one of my favorite Temporal use cases from the project because it was not a long-running cloud service. It was six laptops, a mountain of USB cables, and a physical process that needed reliable coordination for a brief, frantic window. The factory only needed to exist for a day. Failure was not going to respect that schedule.
Temporal managed the badges. Mike managed us.#
The carriers were only one stage of a much larger assembly line.
Badges came out of their shipping packaging. Coworkers attached each screen with VHB tape, fitted the Temporal thumbcap, connected the battery, loaded devices onto hubs, carried full lazy Susans to the flashing stations, unloaded completed badges, returned the empty carriers upstream, attached backplates, and stored the finished units safely for the keynote.
| Our IRL badge distributed system (check out the notorious lazy Susans in the bottom left of the vid.) |
Throughput had to ramp. At first, the assembly stages could not feed six flashing stations continuously. As more badges moved through the line, we shifted people around so one slow stage did not create a pile of half-finished hardware. Mike Waecker stepped in as our emergency assembly project manager and balanced that flow remarkably well.
There is another distributed-systems analogy available here, but I am going to show restraint.
More than 70 Temporalites stepped up to make this happen. They gave us two nights and their one full workday at Replay. They taped screens, attached parts, moved hubs, ran flashing stations, found bottlenecks, and kept 2,300 tiny computers traveling in the right direction. The last badges were finished the night before the morning keynote.
| The last step of the badge assembly process; our charging station. |
| Our team member Melissa with the final product in all its glory! By the way... have you checked out her cafe? I recommend the Durable Execucinno. |
Ignition mattered because it let all of those people help without turning me into a required dependency at every station. The software absorbed the repeatable coordination. I could run one station, step in for real exceptions, and trust coworkers to run the other five without a firmware-flashing seminar.
Edward solved the re-enumeration problem that appeared when the DHL boxes finally gave us QA-firmware hardware again. Mike kept the room from deadlocking on VHB tape or backplates. IT made the stations replaceable. Dozens of coworkers took an emergency that could easily have remained "Shy's badge problem" and made it theirs too.
I am still a little overwhelmed by that.
The control plane around the command#
Ignition did not make esptool obsolete. It did not invent a faster way to write flash memory, and I would not recommend building a Temporal application merely to replace a command that already works.
What Ignition did was wrap that command in the state and policy our production line needed:
- A carrier was an explicit batch with a required device count.
- Every physical badge had an independent, queryable Workflow.
- Hardware operations happened in Activities with heartbeats and bounded logs.
- Retry policy lived with the process instead of in an operator's memory.
- A badge could change operating-system identities without losing its logical identity.
- A successful write was followed by boot and functional validation.
- Each laptop retained enough history to explain its own failures.
- Six isolated stations could scale out without creating a new central dependency.
Ignition has continued evolving since Replay, and the implementation is public in the Replay 2026 Badge repository. The most useful places to explore are the parent and per-badge Workflows, the QA-firmware USB recovery and identity resolution, the boot verification Activity, and the combined-image flashing path.
The public version can flash a single badge on a desk, build from source, or download the latest release factory image. It still uses the same basic shape: a local Temporal server, a batch Workflow, per-badge children, hardware Activities, retries, and validation.
Flashing one badge is still a command.
Flashing 2,300 badges took Temporal, six local servers, 72 carriers' worth of Workflow runs, every lazy Susan at Target, and more than 70 coworkers who decided they were not going to let us fail.
Those 70 coworkers have made me a completely unhelpful outlier on the annual "Would you recommend Temporal as a place to work?" survey.
Ten out of ten.