Temporal Workflows are widely used to build resilient, long-running distributed applications that must finish successfully even if servers crash or networks fail. Common use cases include multi-step business process (e.g. payment processing) automation, durable AI agents, microservice orchestration, and infrastructure automation.
Managing Workflow code changes is unavoidable as applications evolve. This blog shows how you can achieve automated Workflow Versioning with Temporal’s Worker Controller for Worker Versioning and integrate it into your CI/CD pipeline using Github Actions.
Temporal Workflow determinism#
Temporal Workflows are required to be deterministic, which means that given the same input and Event History, your Workflow code must always produce the exact same sequence of commands.
However, as your business processes evolve, it's unavoidable that your Workflow logic will need to change, which can break determinism. Temporal provides a couple of solutions to handle code changes in Temporal Workflows without causing non-determinism errors: Patching and Worker Versioning.
Patching is a code-level branching mechanism inside Workflow definitions that is easy to adopt, but adds clutter to your Workflow source files over time. As more and more changes are introduced, it can become difficult and confusing to manage many different patches.
Worker Versioning manages versions at the infrastructure and deployment level rather than adding version branches directly to your Workflow code.
Worker Versioning#
Worker Versioning allows you to deploy new code changes to Workflows running on your Workers without breaking active executions. Core concepts of Worker Versioning include:
- Worker Deployment groups multiple versions of the same Worker deployment.
- A Worker Deployment Version represents one version within a Worker Deployment and can have multiple Workers running the same build.
- Build ID identifies a single Worker Deployment Version, in combination with a Worker Deployment name.
- A Pinned Workflow is guaranteed to complete on a single Worker Deployment Version.
- An Auto-upgrade Workflow will automatically move to a new code version as you roll it out.
- Each Worker Deployment has a single Current Version which is where Workflows are routed to unless they were previously pinned on a different version.
- Each Worker Deployment can have a ramping version where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version.
Benefits of Worker Versioning include:
- Safe deployments: Prevents non-deterministic errors in long-running Workflows.
- Traffic control: Supports gradual traffic ramping and blue-green or rainbow deployment.
- Instant rollbacks: Quickly reverts traffic to a previous stable version if an issue is detected.
Temporal CLI provides a suite of commands to manage, route, and safely roll out updates to your Worker processes.
- Promoting to Current: Once a new Worker build is verified, you can instantly promote it to be the default receiver for all incoming Workflows.
- Gradual rollouts (ramping): You can direct a small percentage of new Workflows to a newly deployed Worker Deployment Version to verify stability before a full upgrade.
- Workflow pinning & remediation: You can force specific Workflows to lock onto a specific Worker Version, or patch and move stuck Workflows away from buggy builds globally.
However, you might not have access to the CLI in production. Many enterprises lock down the production environment and limit tooling access. I have seen a few of them. In this case, integrating with a CI/CD pipeline is a way to leverage the CLI in a controlled manner.
If manually executing these commands is not for you, the good news is that when deploying your Worker fleets in Kubernetes, you can leverage the Worker Controller to automate all of these.
Worker Controller#
The Temporal Worker Controller is an open-source Kubernetes operator designed to automate, manage, and scale the deployment of Temporal Workers safely. With the Worker Controller, you declare intent in a WorkerDeployment custom resource. The controller reconciles toward it. You can install the controller using the official helm chart.
RELEASE=temporal-worker-controller
NAMESPACE=temporal-system
VERSION=1.0.0
helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \
--version $VERSION \
--namespace $NAMESPACE \
--create-namespace
helm install temporal-worker-controller ./helm/temporal-worker-controller \
--namespace $NAMESPACE \
--create-namespace
Once the operator is deployed, you configure your workloads using custom resources defined by the controller. You must link the controller to your Temporal cluster by defining Worker Options inside your manifests:
connectionRef: Points to a Connection custom resource that holds cluster endpoints and credentials.temporalNamespace: Specifies the target Temporal namespace your Workers will listen to.
Once installed, the controller automates the rest of your production rollout flow:
- Traffic ramping: Pushing a new image tells the controller to launch a new versioned Kubernetes Deployment, progressively moving traffic over based on your chosen rollout strategy.
- Autoscaling templates: You can use a
WorkerResourceTemplatecustom resource to mirror Horizontal Pod Autoscalers (HPAs) safely across all unique Worker versions automatically. - Graceful sunsetting: The controller monitors Temporal to see when old Worker versions are completely drained of pinned workflows, automatically deleting old pods to save costs.
In this demo setup, you’ll see the temporal-system and cert-manager namespaces along with the controller deployments:
GitHub Actions#
GitHub Actions is a CI/CD and automation platform built directly into GitHub. It allows developers to automate, customize, and execute software development processes right inside their code repositories. GitHub Actions function through a hierarchy of components:
-
GitHub Actions workflows: Automated processes defined in YAML files.
-
Events: Specific activities that trigger a GitHub Actions workflow, such as pushing code.
-
Jobs: A set of steps executed sequentially on the same runner.
-
Runners: The virtual machines or servers that execute your jobs.
-
Steps: Individual tasks inside a job that run commands or scripts.
-
Actions: Standalone, reusable commands used within a step. You can build your own actions or use pre-built ones shared on the GitHub Marketplace. Temporal offers a GitHub Action too.
We will integrate Temporal’s Worker Controller with GitHub Actions to achieve automated Worker Versioning!
Setting up automated Worker deployment#
With all the concepts explained, we are ready to put everything together and build our GitHub pipeline to automate the whole versioning process! The ultimate goal is to automatically deploy a Worker Version when new code is pushed in the GitHub repository. Here is what the architecture looks like:
- Code is pushed into the repository.
- The GitHub Actions workflow is triggered.
- Jobs run by the self-hosted runner.
- Docker image is built for the new Worker Version.
- The GitHub Actions workflow applies the
WorkerDeploymentresource, and the Worker Controller creates the versioned Kubernetes Deployment and Worker pods.
The complete code is in this GitHub repository. Key artifacts:
workflows.py- A simple greeting Workflow. Designed to showcase Worker Versioning, when started, it greets, then parks on a signal. Changes to the code triggers GitHub Actions.workerdeployment.template.yaml-The rollout policy. Ramp steps, sunset, pod template.temporal-server.yaml- A Temporal dev server for Minikubedeploy-worker-version.yml- The CI pipeline.
The diagram below shows the high-level process of the pipeline:
See it in action#
Before I explain how everything is built and fit together, let’s see it in action.
- Clone the repo and push it into your own GitHub repository, or fork the repo first then clone it locally into your laptop.
git clone https://github.com/adam-quan/worker-controller-demo.git
-
Because our Kubernetes cluster is in a local Minikube, we need a self-hosted runner. Install and configure the GitHub self-hosted runner by following instructions from your GitHub repository: Settings → Actions → Runners → New self-hosted runner. Make sure you select the appropriate runner image for your OS. Start your runner in a separate terminal.
-
Start a long-running workflow. It greets, then parks on a signal.
source .venv/bin/activate # if not already active
cd worker && python3 starter.py greeting Alice
# started greeting-582cadfb
- Change the workflow. In
worker/workflows.py:
GREETING = "Howdy" # was "Hello"
- Commit the change and push it to the repository, which triggers the GitHub Actions workflow.
git commit -am "change greeting" && git push
- Verify the rollout by starting a new Workflow. It runs on the new code (`
v2-ddb6`):
python3 starter.py greeting Bob
# progress: ['Howdy, Bob! (served by build v2-ddb6)']
The old workflow, still parked, finishes on the version (v1-fb6c) it started on — old greeting, old build — even though v2 is now Current:
python3 starter.py approve greeting-582cadfb
# result: {'greeting': 'Hello, Alice! (served by build v1-fb6c)', 'recorded_by': 'v1-fb6c'}
- Nothing to clean up. Once Alice's workflow finishes, nothing is pinned to the old version any more; Temporal reports it drained and the controller scales it to zero and deletes it, per
spec.sunset.
- In case you need to roll back, simply trigger the
workflow_dispatchaction from your GitHub UI by providing a previously working build ID (image tag), or re-deploy a previously built image tag via thedeploy.shscript:
or
./scripts/deploy.sh v1
The controller recognises a return to a known-good version as a rollback and applies it at once rather than ramping through the steps again. In this demo, executions already running on the bad version are Pinned, so they remain there until they complete.
There you have it! The magic of automated Worker Versioning built into your CI/CD pipeline! Let’s see how all these were built.
The CI pipeline#
The complete pipeline is inside deploy-worker-version.yml. It defines two events that will trigger the pipeline to run, and one deploy job that has a few steps and runs inside the self-hosted runner. The key step is “Build image and update WorkerDeployment”, inside which are the two key deployment steps performed:
- Building the Docker image for the Worker Version with the SHA as the image tag; and
- Deploying the
WorkerDeploymentcustom resource. All the details are in thedeploy.shscript.
docker build -t worker-versioning-demo:$SHA worker/
kubectl apply -f - # the WorkerDeployment, with the new image
GitHub workflow triggering events#
We defined two events that will trigger the GitHub workflow: push and workflow_dispatch:
- The
pushtrigger automatically starts a workflow whenever code is pushed to a repository. The trigger is scoped so that unrelated commits (such as changing theREADME.mdfile) do not redeploy workers. - The
workflow_dispatchtrigger allows you to manually run a workflow on-demand, rather than waiting for automated events like code pushes or pull requests. This allows you to perform a rollback, requiring the image tag as the only input.
on:
push:
branches: [main]
paths:
- "worker/**"
- "k8s/**"
- "scripts/deploy.sh"
workflow_dispatch:
inputs:
image_tag:
description: "Set to a previously built tag to roll back."
Self-hosted runner#
As we mentioned briefly, we need a self-hosted runner. GitHub-hosted runners are ephemeral VMs in GitHub's cloud. They can reach anything on the public internet but not your local Minikube cluster.
jobs:
deploy:
runs-on: self-hosted
A self-hosted runner is a small agent you run on your own machine. It polls GitHub for jobs, executes them locally, and reports back. Because it runs where your tooling already lives, kubectl, Minikube and Docker are simply available. You can download and install the self-hosted runner by following instructions from your GitHub repository: Settings → Actions → Runners → New self-hosted runner.
How versioning is wired up#
In this example, the Git commit SHA forms the first part of the controller-generated Build ID.
- During CI, the Docker images are tagged with SHA of the commit. The SHA was generated when you ran git commit. Git builds a commit object by running a cryptographic hash function over a specific block of text. This allows us to quickly tag our Docker image with a concise commit ID, which will be used as the worker image built ID. It also allows you to see exactly what version of the code is deployed by running this command:
git show BUILD_ID # eg. git show 439744f
- During deployment, the controller injects the worker's identity.
TEMPORAL_DEPLOYMENT_NAMEandTEMPORAL_WORKER_BUILD_IDare set on every container it creates. Note thatTEMPORAL_WORKER_BUILD_IDis derived from the container image tag and hash of the target pod template (configuration hash), and looks like this: “439744f-f8f9”. This is how the Worker deployment looks like from the Minikube dashboard and Temporal Web UI:
You might be wondering why the second part? The second part of the Build ID (f8f9) ensures a unique Build ID even if your application code doesn't change. If you modify a Kubernetes configuration (like updating environment variables or changing memory allocations) but do not push new code changes, the Git SHA (439744f) stays the same, but the configuration hash changes (e.g., to e2a4). This safely triggers a new Worker Version rollout in Temporal without mixing up execution paths.
- Your Worker reads them and registers with them:
worker = Worker(
client,
task_queue=TASK_QUEUE,
workflows=[GreetingWorkflow],
activities=[compose_greeting, record_result],
deployment_config=WorkerDeploymentConfig(
version=WorkerDeploymentVersion(
deployment_name=os.environ["TEMPORAL_DEPLOYMENT_NAME"],
build_id=os.environ["TEMPORAL_WORKER_BUILD_ID"],
),
use_worker_versioning=True,
),
)
Each Workflow declares what should happen to in-flight runs when a newer version becomes Current:
# stays on its original version, forever
@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class GreetingWorkflow: ...
You don’t need the Worker Controller to integrate Worker Versioning into your CI/CD pipeline. Without the Worker Controller, GitHub Actions can invoke the Temporal CLI directly to perform the deployment operations. A separate demo shows that approach for reference.
Summary#
That was a lot to cover! In summary, by integrating Worker Versioning and Worker Controller into your CI/CD pipeline with GitHub Actions, we achieved automated Temporal Workflow Versioning rollout that requires no human intervention.
-
Worker Versioning - Runs multiple code versions side by side and routes each execution to the version that owns it. PINNED Workflows finish where they started; AUTO_UPGRADE Workflows follow the Current version.
-
Worker Controller - Automates Worker deployment. Derives Build IDs, creates a Deployment per version, ramps traffic, promotes, and garbage-collects drained versions.
-
GitHub Actions - Turns a commit into an image and triggers an automated Worker deployment. A self-hosted runner is only needed when the cluster is not publicly reachable.