A durable, egress-only connector for cloud-to-edge messaging - and the two-month setup nightmare that made me build it.
The problem: a cloud app can’t reach a firewalled on-prem edge site - the connection is blocked at the wall, messages are dropped, and the edge is an opaque black box you can’t see into.
I have this memory burned into my mind of the multi-hour long calls every day of the week for a couple of months just to get an application set up before a go-live. Many times those calls were demanded by the customer at the last minute because something else just was not working.
Hours of my time spent on calls with a networking team many time zones away instead of working on the roadmap items that had been planned for my sprint, all because of the complexity it took to make sure messages would be delivered back and forth reliably between an intermediary application, a cloud solution, and an on-premise machine.
I know that AI Agents and AI Native software are all the rage right now, but there is still unglamorous integration plumbing that is needed to make sure warehouses, factories, and other industries operate at scale.
The project I still think about#
Getting prepared for this project’s go live was brutal, and I wish I had known then what I know now, because the solution I landed on stunned me with its simplicity. That’s why I wrote this blog post, and why I’m so excited to share it with you.
When I was an engineer, I was the subject matter expert on our team for all things related to automation tools in a warehouse (think robots that drive pallets around the floor), and that use case came with some fairly complex projects and go-lives, but this one really pushed me to my limits.
It required a major enhancement to a legacy application - the intermediary I mentioned above - which sat between our cloud solution and an edge device that couldn’t connect to the public internet. On top of that, when I inherited it, the application only communicated in one direction. All it did was pull messages from a queue and send those messages to the edge device, so getting it to where it needed to be took real work… code changes to make it bidirectional, a pile of handwritten retry policies to survive dropped messages and flaky connections, and a lot of time spent testing it.
Testing this on my local machine was a pain in the rear end. I could create some dummy applications to be able to make sure that messages were flowing through the process well, but reproducing the real failures - dropped messages off the queue, or a bad connection sending one back to the cloud - was basically impossible. So it required me to actually have to sit down and work with the network team and literally walk them through the steps I needed them to do to run and debug the application for me, since I didn’t have access to what was happening on their network.
From where I sat, their site was a black box.
We all know that talking someone through how to debug your code on a Microsoft Teams screenshare is like pulling teeth. And the worst part is that if something wasn’t working, it required a code rewrite, a new JAR package sent to the customer, and, you guessed it, another Teams call to make sure that the executable was run properly. With only a sliver of overlapping working hours between us, each of those round trips could burn a full day.
It took a toll on me, ate hours of my time each day, and caused a lot of friction on the whole project itself.
Here I am almost a year later still thinking about that project, and thinking about it so much that I decided to build out a ready to go version of it, but with Temporal running as the durable orchestration layer.
I figured I couldn't be the only one who'd been through this. Anyone in a similar spot could use durability, reliability, and visibility out of the box - no crazy defensive logic in the middleware, no DLQs to set up, no inbound firewall rules to negotiate, and plenty more I won't get into here.
What I built instead#
In the old way of processing this, there was a need for some type of queuing system - whether that be SQS, Kafka, BullMQ, etc. - and then a defined polling/consumption mechanism built into the application itself. Then within that mechanism, there was a need for a lot of defensive code to make sure that messages are consumed correctly or that the state is stored in a scenario of failure. That on top of individual requirements per installation meant that there was a lot of plumbing code written and rewritten often.
However, the application that I built is agnostic out of the box and uses Temporal as its durable orchestration layer - replacing the whole “queue + polling + defensive code” stack. The application itself has become a durable middleman that allows for configuration updates from a remote party, and then those configurations are hot-reloaded into the application so that you can configure on the fly even after the application has been installed on a remote network.
So how do you reconfigure a box you can’t even connect to? You flip the direction and have the box connect to you instead. Here, the proxy application keeps a single outbound connection open to the Temporal Service, and that same connection doubles as the control channel - the desired configuration lives in a long-running Temporal Workflow, and when I change it, the update rides back down to the proxy over the link it already holds and hot-reloads in place. This requires no restart, no redeploy, and not a single inbound port opened, which means that the thing that used to take multiple calls with the network team to configure firewall settings or “can you hop on a Teams call right now?” can now be managed remotely over a connection the proxy dialed out itself.
The diagrams below compare the architecture before and after adopting Temporal.
Before Temporal: every durability guarantee lived in custom code you owned - and had to rebuild for every new install.
One proxy per site. Egress-only. Temporal makes every hop durable and de-duplicated.
You can see it running in the companion demo, which ships with a full operations UI for configuring and controlling the proxy live.
What I know, and what I’m still betting on#
Now, I want to be honest that this hasn’t truly been tested in any production environment yet.
I drew on my own experience, worked through a project outline with Claude, read the Temporal documentation, and studied enterprise grade, event driven principles before going back to Claude and bringing the whole thing to life. I’ve tested it on my local machine, Temporal Cloud, and a couple of “dummy” applications to simulate what the real experience could look like.
There could still be some calls with a networking team on the other side of the world to get things set up. There could still be some edge case development to do for your specific scenario, which will require an adequate amount of testing that may or may not be easier (I believe it will be, but that’s still a hypothesis).
But here’s what I do know - no more lost messages, no more hand rolled retry logic between your cloud application and an edge device, and the confidence to tell your customers that your application runs durably end to end.
If you’re the one still stuck#
This is a message to those of you who are still trying to figure out how to durably communicate into a black box.
It is possible, and I am confident that I have laid a durable and visible foundation and pattern for you to build on. This project is open source (MIT license) and available for any purpose, including to fork, make changes to, and run for your own use cases.
I hope it saves you hours of time the way I wish it could have saved me, and I hope it protects your roadmap even more.
I may not be a developer by trade anymore, but boy do I wish I had known about Temporal when I was, because these features coming out of the box would have saved me so much time not only on this project, but countless other ones as well.
If you do choose to use it, please let me know what worked, what didn’t, and what you think needs to be changed.