Securing self-hosted Temporal cross-cluster replication with mTLS

Cross-Cluster (XDC) Replication is a Temporal capability that asynchronously replicates Workflow Executions from active clusters to other passive clusters, for backup and state reconstruction. When necessary, cluster operators can fail over to any of the backup clusters.

Replicating clusters typically sit in different regions and can be connected over the public internet. Securing the replication traffic is almost always a must for production deployments. Mutual TLS (mTLS) encrypts replication traffic and authenticates the connecting peers. Application-level authorization provides an additional layer of access control, with a TokenProvider supplying tokens for outbound cross-cluster requests.

This blog walks through configuring two self-hosted Temporal clusters for cross-cluster replication. We’ll use mTLS to secure client-to-Frontend connections, communication between services within each cluster, and replication traffic between clusters. The same approach extends to more than two clusters. We’ll cover application-level authentication for replication in the next post.

Even though this is written for two clusters, the same process and pattern extends to N clusters.

Architecture overview#

Here is the high-level architecture of the system we will be building throughout the blog. The complete configuration and code are available in this repository.

• Each cluster is a fully independent Temporal deployment: its own Frontend, History, Matching, and Worker services, its own persistence database, and visibility store.
• Replication happens at the application layer: History service ships Workflow history events to the other cluster's History service over a gRPC connection that terminates at each cluster's Frontend service.
• A Namespace is registered as global, listing both clusters. Only one cluster is active for that Namespace at a time; the other is passive/standby.
• TLS layers we need to configure per cluster:
  • Internode - traffic between a cluster's own Frontend/History/Matching/Worker.
  • Frontend - traffic from SDKs, Temporal CLI, and the system Worker into the Frontend service.
  • Remote cluster (replication) - Cluster-to-cluster traffic between the two clusters. This is the piece that specifically secures replication.

We'll use a single self-signed CA to issue certs for both clusters, which is the simplest approach for a self-hosted setup you control end-to-end. In production, you'd typically use per-environment CAs and a real rotation process.

We also implemented JWT token authentication for both clusters, based on the work we have done in the first blog of our three-blog series, to perform Single-Sign-On (SSO) for the Web UI. When accessing the web UI, sign in as temporal-user/temporal.

Generate self-signed CA and certificates#

As mentioned above, we will use a single self-signed CA to issue certs for both clusters. Let’s generate the CA and all the certificates for the two clusters, in three separate directories: certs/ca, certs/cluster-a, certs/cluster-b. Everything we do in this section is encapsulated in the generate-certs.sh script in the repository.

1. Create the root CA. This ca.pem is the single trust anchor every cert below will chain to, and the file every server config will point to for verifying peers.
2. Create a cert for each cluster's internode + frontend traffic. Each cluster needs a cert whose SAN (Subject Alternative Name) covers every hostname other services will use to reach it. This is very important for mTLS, since verification fails on hostname mismatch even if the chain is valid.
3. Generate key + CSR + cert signed by your CA.
4. Create client certs (System Worker + inter-cluster replication client). The System Worker and the replication client connecting to the remote cluster's frontend both need a client cert.
5. After these steps, you’ll have the following keys and certificates:

• certs/ca/ca.key & ca.pem - root CA, trusted by both clusters
• certs/<cluster>/internode.key & internode.pem - internode + internal-frontend identity. Carries serverAuth AND clientAuth, because it is both the certificate the internal frontend presents to the peer cluster and the client certificate this cluster presents when dialing the peer.
• certs/<cluster>/frontend.key & frontend.pem - public frontend identity (serverAuth)
• certs/<cluster>/client.key & client.pem - identity for external callers - the Web UI, the CLI, and SDK clients

6. Mount the directories into the containers to make keys and certificates available for the cluster config.yaml files:

volumes:
  - ./dynamicconfig:/etc/temporal/config/dynamicconfig
  - ./config/cluster-a/config.yaml:/etc/temporal/config/config.yaml:ro
  # Only this cluster's server identities and the CA certificate - never
    	  # the CA key, client keys, or the other cluster's keys.
  - ./certs/ca/ca.pem:/etc/temporal/certs/ca/ca.pem:ro
  - ./certs/cluster-a/internode.pem:/etc/temporal/certs/cluster-a/internode.pem:ro
  - ./certs/cluster-a/internode.key:/etc/temporal/certs/cluster-a/internode.key:ro
  - ./certs/cluster-a/frontend.pem:/etc/temporal/certs/cluster-a/frontend.pem:ro
  - ./certs/cluster-a/frontend.key:/etc/temporal/certs/cluster-a/frontend.key:ro

Both clusters will need ca.pem since it's the shared trust root. Each cluster only needs its own private keys. Never copy private keys across clusters.

Configure TLS in each cluster's config.yaml#

With keys and certificates in place, we can configure Temporal services accordingly. Temporal server TLS configurations live under global.tls which can contain a few different sections:

• The internode section controls backend service (history, matching, internal-frontend) communication TLS settings
• The frontend section controls frontend server TLS settings.
• The systemWorker section controls TLS settings for System Workers connecting to Frontend.
• The remoteClusters section controls the TLS settings for talking to remote clusters.

Of particular interest is the remoteClusters section. It controls outbound TLS for replication traffic. The map is keyed by the host name of the remote cluster (the host part of the address passed to temporal operator cluster upsert), not by its cluster name.

The internode section, the frontend section, and each of the remote cluster sections under remoteClusters all contain a server section and a client section. The server section holds the certificate this cluster or cluster services presents as a client. The client section holds what it trusts in return.

The server contains the following parameters:

• requireClientAuth: Requires clients to authenticate with a certificate when connecting (mTLS).
• certFile: Path to the PEM-encoded certificate.
• keyFile: Path to the PEM-encoded private key.
• clientCaFiles: Paths to the PEM-encoded CA certificates trusted for client authentication.

The client section generally needs to be provided to specify an expected DNS SubjectName contained in the presented server certificate via the serverName field. This is needed as Temporal uses IP to IP communication. The rootCaFiles field also needs to be provided when the client's host does not trust the Root CA used by the server, which is the case for self-signed certificates.

Below is the first cluster’s config. The second cluster's config is similar.

global:
  tls:
    internode:
      client:
        serverName: temporal-a.internal
        rootCaFiles:
          - /etc/temporal/certs/ca/ca.pem
      server:
        requireClientAuth: true
        certFile: /etc/temporal/certs/cluster-a/internode.pem
        keyFile: /etc/temporal/certs/cluster-a/internode.key
        clientCaFiles:
          - /etc/temporal/certs/ca/ca.pem

    frontend:
      client:
        serverName: temporal
        rootCaFiles:
          - /etc/temporal/certs/ca/ca.pem
      server:
        requireClientAuth: true
        certFile: /etc/temporal/certs/cluster-a/frontend.pem
        keyFile: /etc/temporal/certs/cluster-a/frontend.key
        clientCaFiles:
          - /etc/temporal/certs/ca/ca.pem

    remoteClusters:
      # TLS override used specifically for replication traffic TO cluster2
      temporal-b:
        server:
          requireClientAuth: true
          certFile: /etc/temporal/certs/cluster-a/internode.pem
          keyFile: /etc/temporal/certs/cluster-a/internode.key
        client:
          serverName: temporal-b.internal
          rootCaFiles:
            - /etc/temporal/certs/ca/ca.pem

Configure cluster metadata for replication#

To replicate Temporal Workflows across two clusters, we must configure both clusters to recognize each other's metadata, enable global Namespaces, and register Namespaces with the correct failover mapping.

We also configure each cluster's replication in config.yaml. For detailed information, please refer to Temporal’s official documentation.

First cluster:

clusterMetadata:
    enableGlobalNamespace: true
    failoverVersionIncrement: 10
    masterClusterName: "cluster-a"
    currentClusterName: "cluster-a"
    clusterInformation:
        cluster-a:
            enabled: true
            initialFailoverVersion: 1
            rpcName: "frontend"
            rpcAddress: "temporal:7233"
            httpAddress: "temporal:7243"

Second cluster:

clusterMetadata:
    enableGlobalNamespace: true
    failoverVersionIncrement: 10
    masterClusterName: "cluster-b"
    currentClusterName: "cluster-b"
    clusterInformation:
        cluster-b:
            enabled: true
            initialFailoverVersion: 2
            rpcName: "frontend"
            rpcAddress: "temporal-b:7233"
            httpAddress: "temporal-b:7243"

Key points:

• enableGlobalNamespace switches on the replication.
• failoverVersionIncrement must match exactly across all clusters.
• initialFailoverVersion must be unique per cluster and smaller than failoverVersionIncrement.
• rpcAddress points at the frontend service.

Verify TLS#

With the configuration done, let’s verify the TLS connection for both clusters, before connecting them. Start both clusters as if they were standalone (each unaware of the other so far).

1. Generate certificates.

./scripts/generate-certs.sh

2. Bring up just Keycloak, then create the realm, client, role, claim mapper and user that the servers expect. We are still using the Keycloak JWT token for authentication.

docker compose up keycloak -d ./scripts/setup-keycloak.sh

3. Start the first cluster.

docker compose up -d

4. Start the second cluster.

docker compose -f docker-compose.cluster-b.yml up -d

Confirm each cluster is healthy and its TLS is working. You should get a SERVING status from both clusters.

# Health check via CLI, using client identity
docker exec temporal-admin-tools \
  temporal operator cluster health \
  --address temporal:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-a/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-a/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

docker exec temporal-b-admin-tools \
  temporal operator cluster health \
  --address temporal-b:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-b/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-b/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

You can also sanity-check the raw TLS handshake with each cluster directly using openssl:

openssl s_client -connect localhost:7233 \
  -cert ./certs/cluster-a/client.pem \
  -key ./certs/cluster-a/client.key \
  -CAfile ./certs/ca/ca.pem

openssl s_client -connect localhost:8233 \
  -cert ./certs/cluster-b/client.pem \
  -key ./certs/cluster-b/client.key \
  -CAfile ./certs/ca/ca.pem

You should get the message: Verify return code: 0 (ok)

Connect the two clusters#

Now that we know our TLS configuration is working, we can turn on replication by connecting these two clusters. You can run the script connect-clusters.sh to do that.

./scripts/connect-clusters.sh

The script runs temporal operator cluster upsert on cluster-a to register cluster-b, then runs it on cluster-b to register cluster-a.

Verify both sides see each other. You should see both clusters listed with their initialFailoverVersions.

docker exec temporal-admin-tools \
  temporal operator cluster list \
  --address temporal:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-a/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-a/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

docker exec temporal-b-admin-tools \
  temporal operator cluster list \
  --address temporal-b:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-b/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-b/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

The temporal operator cluster list command hits Temporal's Admin Service. The admin service requires System-level Admin privileges (temporal-system:admin). Once again, we are authenticating with our certificates.

The script verify-repliction.sh validates all the configurations of our replication setup with a set of tests.

./scripts/verify-replication.sh

Create a global Namespace#

With replication set up and working, we can create the global Namespace from either cluster, listing both as replication targets and picking the initial active cluster. A global Namespace called replicated was already created when the connect-clusters.sh script was executed. We will manually create one called replicated-namespace here.

docker exec temporal-admin-tools \
  temporal operator namespace create \
  -n replicated-namespace \
  --global \
  --cluster cluster-a \
  --cluster cluster-b \
  --active-cluster cluster-a \
  --address temporal:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-a/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-a/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

Confirm it shows up identically on both clusters:

docker exec temporal-admin-tools \
  temporal operator namespace describe \
  -n replicated-namespace \
  --address temporal:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-a/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-a/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

docker exec temporal-b-admin-tools \
  temporal operator namespace describe \
  -n replicated-namespace \
  --address temporal-b:7233 \
  --tls-cert-path /etc/temporal/certs/cluster-b/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-b/client.key \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem

Both should report the same NamespaceInfo, IsGlobalNamespace: true, and the same active cluster. You can also inspect the Namespace in the Web UI.

Verify replication end to end#

Start a test Workflow Execution on the first cluster, then describe it on the second cluster to confirm that its state has replicated. You can also inspect the execution in the second cluster’s Web UI.

docker exec temporal-admin-tools \
  temporal workflow start \
  -n replicated-namespace \
  --task-queue xdc-demo --type Smoke --workflow-id smoke-1 \
  --address temporal:7233 --tls \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem \
  --tls-cert-path /etc/temporal/certs/cluster-a/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-a/client.key \
  --tls-server-name temporal-a.internal

docker exec temporal-b-admin-tools \
  temporal workflow describe \
  -n replicated-namespace -w smoke-1 \
  --address temporal-b:7233 --tls \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem \
  --tls-cert-path /etc/temporal/certs/cluster-b/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-b/client.key \
  --tls-server-name temporal-b.internal

Once replication catches up, both clusters should report the same run ID.

The repository also includes a Python example that authenticates with a client certificate. The tls_utility.py module creates the TLSConfig passed to Client.connect().

    # The frontend authenticates callers by client certificate
    client = await Client.connect(
        tls_utility.TEMPORAL_ADDRESS,
        namespace=tls_utility.TEMPORAL_NAMESPACE,
        tls=tls_utility.tls_config(),
    )

Test failover#

The verify-replication.sh script also performs a failover test. To flip the active cluster:

docker exec temporal-b-admin-tools \
  temporal operator namespace update \
  -n replicated-namespace \
  --active-cluster cluster-b \
  --address temporal-b:7233 --tls \
  --tls-ca-path /etc/temporal/certs/ca/ca.pem \
  --tls-cert-path /etc/temporal/certs/cluster-b/client.pem \
  --tls-key-path /etc/temporal/certs/cluster-b/client.key \
  --tls-server-name temporal-b.internal

Production hardening notes#

This blog and the associated demo were created for educational purposes. Before deploying to production, address the following:

• Cert rotation: self-signed setups still need a rotation plan. Temporal supports hot-reloading certs on file change in most deployments. Plan to rotate the CA well before its 3650-day expiry and leaf certs before 825 days, with overlap where both old and new certs are trusted during the swap.
• Separate CAs per environment (dev/staging/prod) so a compromised dev cert can't be used against production.
• Least-privilege client certs: the remoteClusters client cert only needs to be trusted for inter-cluster replication. Don't reuse it as a general-purpose SDK client cert.
• numHistoryShards and failoverVersionIncrement are effectively immutable after clusters go live with data. Get these right before onboarding real workloads.
• Monitor replication lag via the replication_tasks_lag metrics Temporal server emits, and alert on it. This is your main signal that mTLS or network issues are silently blocking replication. Consider a proper internal CA instead of a hand-rolled CA once you're past initial setup, so cert issuance and rotation are automated rather than manual openssl runs.

The supported alternative: Temporal Cloud#

This walkthrough shows how to configure and test self-hosted replication with mTLS. Operating it in production also means managing certificates, monitoring replication lag, and testing failover. If you’d prefer Temporal to manage the underlying infrastructure, explore Temporal Cloud’s high availability options.