## Overview [Temporal Workers](https://docs.temporal.io/workers) when run in Kubernetes-based deployments can be easily scaled and managed. [Amazon EKS](https://aws.amazon.com/eks/) (Elastic Kubernetes Service) is a popular choice for running Temporal workers. It integrates seamlessly with other AWS services and provides automatic scaling and fault tolerance, which are essential for production-grade applications. Follow this guide to deploy and manage your Temporal Workers in EKS. You can generally apply this content to both self-hosted Temporal and Temporal Cloud Workflows, but please account for some differences in authentication. For example, you may use mTLS certs instead of API keys when self-hosting Temporal. ## Prerequisites To get started deploying your Workers to EKS, you’ll need: * Your Temporal Cloud account including * [A Namespace using API key authentication](https://docs.temporal.io/cloud/api-keys#namespace-authentication) * Your [API Key for a Service Account](https://docs.temporal.io/cloud/api-keys#generate-an-api-key-for-a-service-account) * An Amazon Web Services (AWS) Account * A deployed EKS cluster within your AWS Account * An installed version of the [aws CLI](https://aws.amazon.com/cli/) * [docker](https://www.docker.com/get-started/) * The [kubectl](https://kubernetes.io/docs/reference/kubectl/) command line tool, configured with your deployed EKS cluster ## Getting Started This guide walks you through writing Temporal Worker code, containerizing and publishing the Worker to the Amazon Elastic Container Registry (ECR), and deploying the worker to Amazon EKS. The example in this post uses Temporal’s [Python SDK](https://github.com/temporalio/sdk-python). ### Write the Worker Code In Temporal applications, business logic lives within your main [Workflow](https://docs.temporal.io/workflows) code. Your Worker code runs separately, and is responsible for executing your Workflows and Activities. Make sure to configure your Worker to use environment variables so you can dynamically route your Worker to different Temporal Instances, Namespaces, and Task Queues on the fly: ```python TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233") TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default") TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue") TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "") ``` After configuration, instantiate your Temporal client: ```python client = await Client.connect( TEMPORAL_ADDRESS, namespace=TEMPORAL_NAMESPACE, rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE}, api_key=TEMPORAL_API_KEY, tls=True ) ``` Now, instantiate and run your Worker: ```python worker = Worker( client, task_queue=TEMPORAL_TASK_QUEUE, workflows=[your_workflow], activities=[ your_first_activity, your_second_activity, your_third_activity ] ) await worker.run() ``` Here is a complete Python boilerplate that showcases how to instantiate a Client and pass it to the Worker before starting the Worker execution: ```python import asyncio import os from temporalio.worker import Worker from temporalio.client import Client from workflows import your_workflow from activities import your_first_activity, your_second_activity, your_third_activity TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233") TEMPORAL_NAMESPACE = os.exxxnviron.get("TEMPORAL_NAMESPACE", "default") TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue") TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "you-api-key") async def main(): client = await Client.connect( TEMPORAL_ADDRESS, namespace=TEMPORAL_NAMESPACE, rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE}, api_key=TEMPORAL_API_KEY, tls=True ) print("Initializing worker...") # Run the worker worker = Worker( client, task_queue=TEMPORAL_TASK_QUEUE, workflows=[your_workflow], activities=[ your_first_activity, your_second_activity, your_third_activity ] ) print("Starting worker... Waiting for tasks.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ### Containerize the Worker for Kubernetes You need to containerize your Worker code to run it with [Kubernetes](https://kubernetes.io/). Here is a sample Python Dockerfile, complete with the Temporal Python SDK installed: ``` # Use Python 3.11 slim image as base FROM python:3.11-slim # Set working directory WORKDIR /app # Install system dependencies RUN apt-get update && apt-get install -y \ gcc \ && rm -rf /var/lib/apt/lists/* # Install the Temporal Python SDK dependency RUN pip install --no-cache-dir temporalio # Copy application code COPY . . # Set Python to run in unbuffered mode ENV PYTHONUNBUFFERED=1 # Run the worker CMD ["python", "worker.py"] ``` Build the Docker image and target the `linux/amd64` architecture: ``` docker buildx build --platform linux/amd64 -t your-app . ``` ### Publish the Worker Image to Amazon ECR After building the Docker image, you’re ready to publish it to Amazon ECR . Make sure that you’re authenticated with AWS, and that you’ve set your `AWS_REGION` and `AWS_ACCOUNT_ID` environment variables: ``` export AWS_ACCOUNT_ID= export AWS_REGION= ``` Create an ECR repository and authenticate ECR with the Docker container client: ``` aws ecr create-repository --repository-name your-app aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com ``` After authenticating Docker with ECR, tag your container and publish it: ``` docker tag your-app $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest ``` ### Deploy the Workers to EKS With your Worker containerized, you’re ready to deploy it to EKS. Create a namespace in your EKS cluster. You’ll use the namespace to run your Temporal Workers: ``` kubectl create namespace your-namespace ``` Create a `ConfigMap` to hold non-sensitive values that Kubernetes will injectinto your Worker deployment. These enable dynamic routing for instances, Namespaces, and Task Queues. Create a `config-map.yaml` file like the following example, which sets these values: ``` apiVersion: v1 kind: ConfigMap metadata: name: temporal-worker-config namespace: temporal-system data: TEMPORAL_HOST_URL: ““ TEMPORAL_NAMESPACE: “” TEMPORAL_TASK_QUEUE: “” ``` Apply the `ConfigMap` to your namespace: ``` kubectl apply -f config-map.yaml \ --namespace your-namespace ``` For sensitive values, use [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). Create a secret to hold your Temporal API key: ``` kubectl create secret generic temporal-secret \ --from-literal=TEMPORAL_API_KEY=$TEMPORAL_API_KEY \ --namespace your-namespace ``` With your configuration in place, you can deploy the Worker. Create a `deployment.yaml` file configuring your Worker image, resources, and secret values. In normal deployment, tune your `resources` for production workloads. Note that the spun-up container reads your Temporal API key from the Kubernetes secret you just created: ``` apiVersion: apps/v1 kind: Deployment metadata: name: your-app namespace: your-namespace labels: app: your-app spec: selector: matchLabels: app: your-app replicas: 1 template: metadata: labels: app: your-app spec: serviceAccountName: your-app containers: - name: your-app image: env: - name: TEMPORAL_ADDRESS valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_ADDRESS - name: TEMPORAL_NAMESPACE valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_NAMESPACE - name: TEMPORAL_TASK_QUEUE valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_TASK_QUEUE - name: TEMPORAL_API_KEY valueFrom: secretKeyRef: name: temporal-secret key: TEMPORAL_API_KEY resources: limits: cpu: "0.5" memory: "512Mi" requests: cpu: "0.2" memory: "256Mi" ``` Apply the `deployment.yaml` file to the EKS cluster: ``` kubectl apply -f deployment.yaml \ --namespace your-namespace ``` ### Verify the Workers are Connected After deploying your Workers to EKS, confirm that they have connected to Temporal Cloud. Retrieve the pod listing for the Kubernetes/EKS namespace that you created: ``` kubectl get pods -n temporal-system ``` After listing the pods, access the Worker logs to confirm you’re properly connected to Temporal Cloud: ``` kubectl logs -n temporal-system ``` You confirm connection when you see: ``` Initializing worker... Starting worker... Waiting for tasks. ``` You have now successfully deployed your Temporal Worker to EKS. Deployment, however, is only a part of the lifecycle of your workers! Stay tuned for future guides about auto-scaling your Temporal workers.

Resources > Replay 2024

Keynote: Building Trust With Code

Replay 2024 Video Library