Quick launch Temporal Workers on Amazon Elastic Kubernetes Service (EKS)

Overview

Temporal Workers when run in Kubernetes-based deployments can be easily scaled and managed. Amazon EKS (Elastic Kubernetes Service) is a popular choice for running Temporal workers. It integrates seamlessly with other AWS services and provides automatic scaling and fault tolerance, which are essential for production-grade applications.

Follow this guide to deploy and manage your Temporal Workers in EKS. You can generally apply this content to both self-hosted Temporal and Temporal Cloud Workflows, but please account for some differences in authentication. For example, you may use mTLS certs instead of API keys when self-hosting Temporal.

Prerequisites

To get started deploying your Workers to EKS, you’ll need:

Your Temporal Cloud account including
- A Namespace using API key authentication
- Your API Key for a Service Account
An Amazon Web Services (AWS) Account
- A deployed EKS cluster within your AWS Account
An installed version of the aws CLI
docker
The kubectl command line tool, configured with your deployed EKS cluster

Getting Started

This guide walks you through writing Temporal Worker code, containerizing and publishing the Worker to the Amazon Elastic Container Registry (ECR), and deploying the worker to Amazon EKS. The example in this post uses Temporal’s Python SDK.

Write the Worker Code

In Temporal applications, business logic lives within your main Workflow code. Your Worker code runs separately, and is responsible for executing your Workflows and Activities. Make sure to configure your Worker to use environment variables so you can dynamically route your Worker to different Temporal Instances, Namespaces, and Task Queues on the fly:

TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")
TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default")
TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")
TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "")

After configuration, instantiate your Temporal client:

client = await Client.connect(
    TEMPORAL_ADDRESS,
    namespace=TEMPORAL_NAMESPACE,
    rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE},
    api_key=TEMPORAL_API_KEY,
    tls=True
)

Now, instantiate and run your Worker:

worker = Worker(
    client,
    task_queue=TEMPORAL_TASK_QUEUE,
    workflows=[your_workflow],
    activities=[
        your_first_activity,
        your_second_activity,
        your_third_activity
    ]
)

await worker.run()

Here is a complete Python boilerplate that showcases how to instantiate a Client and pass it to the Worker before starting the Worker execution:

import asyncio
import os

from temporalio.worker import Worker
from temporalio.client import Client

from workflows import your_workflow
from activities import your_first_activity, your_second_activity, your_third_activity

TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")
TEMPORAL_NAMESPACE = os.exxxnviron.get("TEMPORAL_NAMESPACE", "default")
TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")
TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "you-api-key")

async def main():
	client = await Client.connect(
		TEMPORAL_ADDRESS,
		namespace=TEMPORAL_NAMESPACE,
		rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE},
		api_key=TEMPORAL_API_KEY,
		tls=True
	)

print("Initializing worker...")
	# Run the worker
	worker = Worker(
		client,
		task_queue=TEMPORAL_TASK_QUEUE,
		workflows=[your_workflow],
		activities=[
			your_first_activity,
			your_second_activity,
			your_third_activity
		]
	)

	print("Starting worker... Waiting for tasks.")
	await worker.run()

if __name__ == "__main__":
	asyncio.run(main())

Containerize the Worker for Kubernetes

You need to containerize your Worker code to run it with Kubernetes. Here is a sample Python Dockerfile, complete with the Temporal Python SDK installed:

# Use Python 3.11 slim image as base
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    && rm -rf /var/lib/apt/lists/*

# Install the Temporal Python SDK dependency
RUN pip install --no-cache-dir temporalio

# Copy application code
COPY . .

# Set Python to run in unbuffered mode
ENV PYTHONUNBUFFERED=1

# Run the worker
CMD ["python", "worker.py"]

Build the Docker image and target the linux/amd64 architecture:

docker buildx build --platform linux/amd64 -t your-app .

Publish the Worker Image to Amazon ECR

After building the Docker image, you’re ready to publish it to Amazon ECR . Make sure that you’re authenticated with AWS, and that you’ve set your AWS_REGION and AWS_ACCOUNT_ID environment variables:

export AWS_ACCOUNT_ID=<your_aws_account_id>
export AWS_REGION=<your_aws_region>

Create an ECR repository and authenticate ECR with the Docker container client:

aws ecr create-repository --repository-name your-app
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

After authenticating Docker with ECR, tag your container and publish it:

docker tag your-app $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest

Deploy the Workers to EKS

With your Worker containerized, you’re ready to deploy it to EKS. Create a namespace in your EKS cluster. You’ll use the namespace to run your Temporal Workers:

kubectl create namespace your-namespace

Create a ConfigMap to hold non-sensitive values that Kubernetes will injectinto your Worker deployment. These enable dynamic routing for instances, Namespaces, and Task Queues. Create a config-map.yaml file like the following example, which sets these values:

apiVersion: v1
kind: ConfigMap
metadata:
  name: temporal-worker-config
  namespace: temporal-system
data:
  TEMPORAL_HOST_URL: “<your-temporal-address>“
  TEMPORAL_NAMESPACE: “<your-temporal-cloud-namespace>”
  TEMPORAL_TASK_QUEUE: “<your-task-queue>”

Apply the ConfigMap to your namespace:

kubectl apply -f config-map.yaml \
    --namespace your-namespace

For sensitive values, use Kubernetes Secrets. Create a secret to hold your Temporal API key:

kubectl create secret generic temporal-secret \
    --from-literal=TEMPORAL_API_KEY=$TEMPORAL_API_KEY \
    --namespace your-namespace

With your configuration in place, you can deploy the Worker. Create a deployment.yaml file configuring your Worker image, resources, and secret values. In normal deployment, tune your resources for production workloads. Note that the spun-up container reads your Temporal API key from the Kubernetes secret you just created:

apiVersion: apps/v1
kind: Deployment
metadata:
   name: your-app
   namespace: your-namespace
   labels:
      app: your-app
spec:
   selector:
      matchLabels:
         app: your-app
   replicas: 1
   template:
      metadata:
         labels:
            app: your-app
      spec:
         serviceAccountName: your-app
         containers:
            - name: your-app
              image: <your-ecr-image-name>
              env:
                - name: TEMPORAL_ADDRESS
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_ADDRESS
                - name: TEMPORAL_NAMESPACE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_NAMESPACE
                - name: TEMPORAL_TASK_QUEUE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_TASK_QUEUE
                - name: TEMPORAL_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: temporal-secret
                      key: TEMPORAL_API_KEY
              resources:
                limits:
                  cpu: "0.5"
                  memory: "512Mi"
                requests:
                  cpu: "0.2"
                  memory: "256Mi"

Apply the deployment.yaml file to the EKS cluster:

kubectl apply -f deployment.yaml \
    --namespace your-namespace

Verify the Workers are Connected

After deploying your Workers to EKS, confirm that they have connected to Temporal Cloud. Retrieve the pod listing for the Kubernetes/EKS namespace that you created:

kubectl get pods -n temporal-system

After listing the pods, access the Worker logs to confirm you’re properly connected to Temporal Cloud:

kubectl logs <pod-name> -n temporal-system

You confirm connection when you see:

Initializing worker...
Starting worker... Waiting for tasks.

You have now successfully deployed your Temporal Worker to EKS. Deployment, however, is only a part of the lifecycle of your workers! Stay tuned for future guides about auto-scaling your Temporal workers.