As internet software grows more sophisticated, automated, and autonomous, it is becoming more asynchronous and longer-running. Longer-running Workflows can hop between multiple processes, making them resilient after process shutdowns and code bugs, and able to scale different steps independently.
Thinking beyond single processes presents challenges related to sleeping, checkpointing, and retries, which Temporal Workflows make significantly easier. Most often, the remaining complaint is that it’s hard to deal with Workflow Versioning — cases when two processes are running two different versions of your code.
With our new feature, Worker Versioning, you can now virtually eliminate version compatibility problems for most Workflows by guaranteeing that each Workflow will run on a single version of your code. We call this Workflow Pinning.
With Pinned Workflows, you don’t have to think about interface compatibility between your Workflows and Activities, nor versioning problems when your Workflow code switches from one version to another.
To enable this, you’ll need a good deployment strategy to manage your Workers. It’s worth taking a step back to talk about this — what are deployment strategies and how do you pick the right one?
I’ll define and weigh three strategies. From least to most robust, they are called rolling, blue-green, and rainbow deploys. Rainbow deploys aren’t widely known today, but over time, they will become standard practice for high-availability, asynchronous, multi-process Workflows.
Rolling the dice with rolling deploys#
In a rolling deploy, Worker nodes are gradually cycled in place from the old version to the new version. These have limitations for high-availability services — even synchronous services. When something goes wrong with the new code being deployed, it can take a long time to roll back to the previous version.
Another notable issue with this process is the lack of control over where your Workflows are routed, making your rollouts inherently unsafe.
Mean, lean, blue-green machine#
During a Worker deployment, a blue-green strategy temporarily rolls out a second “green” version of your Workers while keeping the old “blue” one around. With two available copies, you can route traffic to each version as you see fit. There are several major reliability advantages of this. It allows you to:
- Easily run tests on a new version before cutting over traffic.
- Ramp up traffic at a pace you choose.
- Instantly roll back when something is wrong.
Blue-green deploys are a big win for synchronous and asynchronous services alike. Still, there’s a problem with blue-green deploys for long-running multi-process Workflows — there are only two colors! This brings us to rainbow deploys.
Taste the rainbow#
Rainbow deploys can route like blue-green deploys, but to more colors.
To pin Workflows to a code version, you must be able to track the usage of old versions and keep them around until all the Workflows complete, i.e., the version is drained. Doing this with high availability requires rainbow deploys. That’s because, when you ship a bug, or if you want to deploy twice in quick succession, it’s easy to get into situations where you need to keep three or more builds around.
Worker Versioning helps you set up rainbow deploys, and Temporal will track when your Worker versions have drained.
Now that I’ve introduced the three deployment strategies, let me talk about how Temporal enables seamless revisions of Workflows through blue-green and rainbow deployments.
Introducing Worker Versioning#
We embrace our asynchronous, long-running future with Worker Versioning, a powerful new tool for managing your Worker deployments. The magic comes as you maintain your Workflows. Each time you declare a Workflow type, you’ve got two options.
- Pinned Workflows stay on a single version through their lifetimes, removing the need for you to think about versioning or interface compatibility between your Activities and Workflows. Declare your Workflow type as pinned like so, and you can freely make changes without affecting old Workflows (unless you want to affect old Workflows).
@workflow.defn(versioning_behavior=common.VersioningBehavior.PINNED)
class MyPinnedWorkflow:
@workflow.run
async def run(self) -> None:
workflow.logger.info(
"Whee! I can make changes without worrying about versioning!")
- Auto-upgrade Workflows automatically hop to a new version as part of a blue-green or rainbow deployment, benefitting from pre-deployment testing, gradual ramp-ups, and instant rollbacks. They still should be patched when modifications have been made, but it is easier to catch patching bugs early. When using rainbow deploys, you can tag Workflows as Auto-upgrade when they will run longer than you’d like to keep versions of your code around.
@workflow.defn(versioning_behavior=common.VersioningBehavior.AUTO_UPGRADE)
class AutoUpgradingWorkflow:
@workflow.run
async def run(self) -> None:
workflow.logger.info(
"I need to patch if I make changes, but I can do it more safely with" +
"blue-green or rainbow deploys!")
What makes this work? Temporal now provides, in both open source and Cloud:
- A built-in task router to help you instantly cut between versions or gradually ramp them up. (This substitutes for a traditional load balancer that you would use for blue-green deploys of a synchronous service.)
- First-class support for rainbow deployments which allows you to pin Workflows to a single version of your Workers, eliminating most version and interface compatibility problems.
- A fully supported Worker Controller for Kubernetes, co-developed with Jacob LeGrone at DataDog, as an easy way to adopt rainbow deployments.
Adopting Worker Versioning will let you change your applications safely, confidently, and with higher ambition.
In the remainder of this post, I’ll describe:
- Building Worker Versioning — our design philosophy and the challenges we faced.
- Getting started with Worker Versioning — links to get you started.
- Project status — what we have and what’s to come.
Building Worker Versioning#
Our four overriding design goals were ease of adoption, reliability, deployment safety, and flexibility.
Ease of adoption#
Application developers tell us every day that they don’t want to deal with Workflow versioning problems. However, to have nice things, they often need to upgrade deployment systems that another team at the company might manage. To help such teams, we obsessed over ease of adoption:
- We believe rainbow deploys are the future and want to make them really easy to adopt, which is why we polished the APIs to make them as simple as possible, and also created the Temporal Worker Controller for Kubernetes as a drop-in solution.
- We know that blue-green is a more established deployment strategy in the industry, so we added some features to unlock the value of blue-green deploys independently of rainbow deploys.
Reliability#
With Worker Versioning, task queues can now be routed based on version. This is a fundamental change to Temporal’s architecture, so we wanted to make sure we “nail it and scale it.” To this end, we’ve been dogfooding it within Temporal Cloud for several months, as well as stress testing it.
Deployment safety#
We prioritized making deployments as safe as possible:
- Code revisions shouldn’t affect old Workflows. When they do, users should be able to use testing and controlled rollouts to minimize damage.
- Zero downtime is the goal. This meant providing features like instant rollback, pre-deployment testing, and tracking when versions are drained of Workflows.
- Recovery is always possible — we never want users to get stuck.
Flexibility#
We know customers have a great variety of systems set up to manage Worker deployments, and a variety of Workflow patterns. We provided primitives for managing diverse deployments, Workflows of varying run lengths, and the ability to override default task routing. We plan to add more flexibility over time based on demand — please tell us about your scenarios.
Getting started with Worker Versioning#
To get started with Worker Versioning, you have a few options:
- Integrate directly with our APIs and commands. Start here.
- Use the Worker Controller for Kubernetes.
- We’ve also recently released a free, self-paced Worker Versioning course.
Project status and roadmap#
Both Worker Versioning and our Worker Controller are in public preview, which means they are recommended for staging and for production usage after careful testing.
We need feedback! Let us know what’s working and what else you need. This will help make our GA great. Find us in #safe-deploys in our Community Slack.
Worker Versioning plans#
- A light refresh is due at the end of September with some improvements.
- We expect to make it generally available (GA) in Q4 2025.
- We plan to also add a third Workflow type annotation, Pinned-until-Continue-as-New, for very long-running Workflows. They will stay pinned and be awoken when it’s time to continue as new. Dates are TBD.
- We plan to make it more scalable to move broken Workflows en masse to a patch build, and also to add replay testing. Dates are TBD.
Worker Controller plans#
- As of September 2025, our next priority is auto-scaling. With rainbow deploys, we know it’s particularly important to scale down old builds as they drain to ease resource usage.
- Please let us know if you would like support for other deployment infrastructure besides Kubernetes.
Get in touch#
We want to hear from you! Find #safe-deploys in our Community Slack to join the conversation and to keep up with news about Worker Versioning.