How Temporal powers workflows at HeyGen

This is a guest post from Jiajun Zhao, a software engineer at HeyGen.

Generating a HeyGen video requires coordination among many services. A script is converted to speech, timing data is extracted, assets are prepared, avatar scenes run through GPU inference, graphics are rendered, and the results are composited, captioned, transcoded, and delivered.

The workflow must also keep billing accurate, enforce product limits, report useful progress, and recover cleanly when a service or worker fails. Traffic bursts from one customer should not delay everyone else, and a failure late in the pipeline should not leave either the video or the customer's credits in an inconsistent state.

This makes video generation an orchestration problem as much as a media-processing problem.

Today, Temporal coordinates these workflows across avatar video, video translation, avatar creation, voice, and rendering. At our current scale, we run millions of workflow executions and tens of millions of activity executions each day, spanning roughly 300 workflow types, nearly 1,000 registered activities, around 100 task queues, and more than 50 independently scaled worker deployments. GPU workers run across several clouds and serverless GPU providers.

This post covers why our previous system became difficult to operate, how we model video generation in Temporal, the shared platform capabilities we built for product teams, and how we scale and utilize a heterogeneous GPU fleet.

The system we outgrew#

Before Temporal, we operated an internal job system.

The system combined a MySQL job table, a polling scheduler, RabbitMQ, Celery workers, and a separate service for tracking job completion:

Figure 1. Legacy job orchestration before Temporal.

When a video was submitted, the API wrote the entire job graph to MySQL. The scheduler repeatedly searched for runnable jobs and sent them to RabbitMQ. Workers reported their status through HTTP callbacks, and the tracker updated the database, activated downstream jobs, and ran product-specific completion logic.

This architecture supported HeyGen through several stages of growth. Over time, however, its correctness increasingly depended on the database, broker, workers, scheduler, and callback service remaining synchronized.

A worker might complete a job but fail to deliver its callback, or it might stop before sending a final status. A tracker restart could drop requests. An interruption in the database or broker could leave different parts of the system with different views of the same job.

From the user's perspective, these failures usually looked the same: a video remained at a particular progress percentage indefinitely. From the on-call perspective, recovery often meant inspecting job rows, reconstructing what had happened, correcting state, and manually requeueing work.

The architecture also slowed product development. Workflow logic was divided among the API code that created jobs, the scheduler that decided when they could run, and the tracker that handled completion. Adding a pipeline step often required changes in all three places. Local testing required personal queues and callback configuration, while product teams implemented their own variations of retries, refunds, and rate limits.

During one managed RabbitMQ upgrade, work stalled across the platform. We brought up replacement capacity and repaired affected jobs manually. The incident was an especially visible example of a recurring problem: workflow state was spread across several systems, and recovery depended on reconciling them after a failure.

We needed an execution model in which workflow state, retries, and recovery were part of the same system.

Modeling video generation as a Temporal Workflow#

The mapping from our old model to Temporal was straightforward: a flow became a Workflow, and a job became an Activity.

The important difference was that the graph no longer had to be written to a table and reconstructed by a polling scheduler. The Workflow could express sequencing, parallelism, compensation, and completion directly in code, while Temporal persisted its execution history.

A simplified version of the main video Workflow looks like this:

@workflow.defn(name=WorkflowType.HEYGEN_VIDEO)
class HeygenVideoWorkflow(BaseWorkflow):
    @workflow.run
    async def run(
        self,
        input: HeygenVideoWorkflowInput,
    ) -> HeygenVideoWorkflowOutput:
        return await self.run_activities(input)

    async def run_activities(self, input):
        try:
            # TTS establishes the actual generated duration.
            draft = await execute_activity(
                Act.tts,
                ...,
                queue=VOICE_CPU,
            )

            await execute_activity(Act.check_video_limits, ...)
            plan = await execute_activity(Act.draft_to_plan, ...)

            # Charge only after duration and render work are known.
            await execute_activity(Act.deduct_quota, ...)

            # Render avatar scenes concurrently with graphics.
            await asyncio.gather(
                *(
                    self._render_avatar_scene(scene)
                    for scene in plan.avatar_scenes
                ),
                self._render_graphics(plan),
            )

            await execute_activity(
                Act.render_and_composite,
                ...,
                queue=RENDER_GPU,
            )

            return await self._complete(input)

        except Exception as exc:
            await self._refund_if_charged(input)
            await self._mark_failed_and_notify(input, exc)
            raise

The production Workflow coordinates close to 90 distinct activity types. Its high-level structure is:

Figure 2. High-level structure of the video-generation workflow.

When an Activity worker stops during execution, Temporal can retry the Activity on another worker according to its retry policy. When a Workflow worker stops, another worker can reconstruct the Workflow from its persisted event history and continue execution.

We no longer need a separate callback protocol to reconcile the worker's state with the scheduler's state. The workflow history records what was scheduled, what completed, what failed, and what is still waiting.

Keeping scene orchestration in one workflow#

A typical video contains several scenes, and each avatar scene moves through preprocessing, GPU inference, and postprocessing.

We implement these scene pipelines as concurrent coroutines inside the main workflow rather than creating a child workflow for every scene:

async with sems.pre_processing_semaphore:
    inference_input = await self._pre_process_scene(scene)

async with sems.inference_semaphore:
    inference_output = await avatar_workflow.run_inference_activities(
        inference_input
    )

async with sems.post_processing_semaphore:
    await self._post_process_scene(inference_output)

Each stage has its own concurrency limit. While one scene is using a GPU, another can prepare its inputs and a third can complete postprocessing. We generally schedule longer scenes first, and some models combine short scenes into bounded inference batches to reduce dispatch overhead.

Keeping this work in one workflow gives us a single, readable execution history for the video. We use child workflows selectively for operations that need an independent timeout, retry boundary, or failure domain. For example, an external canvas-rendering operation has its own execution timeout, preventing it from waiting indefinitely and blocking the parent video workflow.

Applying fairness at multiple layers#

Temporal provides durable execution and task dispatch, but customer-level fairness remains application logic.

We built a scheduling and admission layer alongside Temporal to enforce concurrency limits based on product entitlements, workspace quotas, and current usage. The scheduler controls how much work each customer can introduce into the system at one time, preventing a single workspace or traffic burst from consuming a disproportionate share of GPU capacity.

Once activities have been scheduled onto a Temporal task queue, the priority assigned to each task determines dispatch order when workers are busy.

The two controls address different parts of the problem:

• Our scheduler controls admission and concurrency: how much work each customer may have active or admit into execution at a given time.

• Temporal task priority controls dispatch order: which already-admitted tasks workers receive first when capacity is constrained.

Temporal gives us the mechanisms for durable execution and prioritized dispatch. We own the policy that maps subscriptions, quotas, and product entitlements onto those mechanisms.

The shared workflow platform#

Once several products were using Temporal, we built a shared execution layer so that each team would not have to create its own conventions.

The goal is for product engineers to focus on two things: implementing activities and writing the workflows that coordinate them. Worker configuration, identity propagation, error handling, metrics, visibility, and development tooling come from the platform.

Reusing workflow modules#

Our workflow base class separates the Temporal entry point from a deterministic run_activities() orchestration method:

@workflow.run
async def run(self, input):
    return await self.run_activities(input)

This lets us use the same workflow logic in several ways:

• As a top-level workflow

• As a child workflow with its own history and lifecycle

• As an inlined module within another workflow

Inlining a workflow module means calling its deterministic orchestration method directly. The activities it schedules become part of the caller's history without introducing a separate child execution.

We use this pattern for components such as avatar inference, moderation, and media processing when they do not require their own failure or timeout boundary. We use child workflows when isolation is worth the additional execution history and lifecycle management.

The distinction keeps composition simple: modules by default, and child workflows when they provide a specific operational benefit.

Centralizing cross-cutting behavior#

Every worker installs a common interceptor stack. The interceptors provide behavior that should remain consistent across products:

• Identity propagation. User, workspace, subscription, and canary context are added to activity and child-workflow inputs automatically.

• Error classification. Internal errors are translated into Temporal application errors using one shared retryability policy. Transient server-side and rate-limit errors are retryable, while user-input errors do not consume repeated attempts.

• Metrics. The platform records activity and workflow throughput, latency, failures, and compute usage. One useful metric compares seconds of computation with seconds of generated video.

• Search attributes. Workflow metadata such as customer identity, subscription tier, and the currently failing activity is recorded in Temporal visibility, allowing support and on-call engineers to find affected executions.

• Heartbeat defaults. Long-running activities receive standard heartbeat behavior, and GPU activities can use an automatic heartbeater so that a lost worker is detected without each team implementing its own loop.

Because these concerns are centralized, workflow code remains focused on orchestration rather than repeating operational boilerplate.

Making workers self-service#

We operate separate worker deployments for workloads with different resource and scaling requirements. A GPU worker that keeps a large model in memory should not share a deployment with CPU-based media processing or delivery activities.

A scaffolding tool creates the task-queue configuration, deployment definition, and CI integration for a new worker. Queue names can be overridden by environment, giving developers personal task queues for local work. In a development environment, a workflow input can route a selected activity to a local queue, allowing an engineer to test one part of a larger workflow on a laptop.

We also provide an activity harness that runs a registered activity in-process from a JSON payload. This gives engineers a simpler way to test activity behavior without creating a complete workflow execution.

Finally, CI checks enforce several architectural rules. Workflow files may import activity interfaces but not their implementations, and automated checks prevent known nondeterministic operations from entering workflow code.

Scaling a heterogeneous GPU fleet#

GPUs are expensive and are a major cost for an AI company. This section covers briefly how we manage GPU autoscaling and improve GPU utilization across activities to improve cost efficiency and utilization.

Each major workload has its own task queue, worker deployment, and scaling policy. This isolates workloads with different resource requirements and prevents one queue's behavior from affecting every worker rollout.

CPU workers run in Kubernetes and scale based on Temporal task-queue backlog through KEDA. GPU workers are more complicated because our capacity is distributed across several GPU clouds and serverless GPU providers.

Temporal's pull-based worker model makes the multi-provider setup relatively simple. Workers make outbound connections to Temporal and poll a task queue. Adding capacity in another environment usually means starting the same worker image with the appropriate configuration and queue name. It does not require exposing a new inference service or extending a request-routing layer.

The harder problem is deciding how much capacity to run in each environment.

Why independent autoscalers were not enough#

Our first approach used a separate autoscaler in each Kubernetes cluster, along with independent scaling for serverless providers.

All of these controllers watched the same task-queue backlog. When traffic increased, several controllers could react at the same time and provision more capacity than the queue required. They also had no shared understanding of cost or allocation preferences, such as using committed Kubernetes capacity before adding serverless instances.

We replaced those independent decisions with a global autoscaling coordinator. The coordinator itself is implemented as a Temporal workflow.

A long-running workflow coordinates each GPU task queue. At regular intervals, it collects demand and capacity data, calculates a target allocation, and applies the required changes. It periodically calls continue_as_new() to keep its history bound.

Figure 3. Global autoscaling coordinator for a GPU task queue.

Three parts of this design are particularly important.

Measuring worker readiness#

Every GPU worker periodically reports one of three states: starting, idle, or busy.

Infrastructure-level readiness is not enough for our workloads. Kubernetes may report that a pod is running before its model has finished loading into GPU memory. Temporal may know that a worker has begun polling, but not whether that worker is ready to start inference.

By counting model-loading workers as warming capacity, the autoscaler avoids responding to the same demand twice. A traffic increase that occurs while new workers are starting does not immediately trigger another redundant scale-up.

Scaling up quickly and scaling down carefully#

Scale-up is based partly on rate imbalance: whether tasks are arriving faster than the fleet can dispatch them. The coordinator estimates the additional capacity required from observed throughput and includes headroom for short-term variation.

Scale-down requires sustained low utilization rather than a single quiet interval. We use separate scale-up and scale-down thresholds, cooldown windows, and configuration checks intended to prevent frequent oscillation.

The result is deliberately asymmetric. Insufficient capacity affects user latency immediately, while excess capacity can be removed more gradually.

Allocating capacity by cost and preference#

Deployments are ordered by allocation priority.

The coordinator fills committed or lower-cost Kubernetes capacity first. It then adds serverless capacity when demand exceeds those limits. During scale-down, it removes higher-cost burst capacity first.

This also gives us a straightforward way to handle provider disruptions. Capacity can be shifted by changing allocation limits or priorities rather than redesigning how requests are routed.

Early rollout comparisons showed materially less over-provisioning than the previous independent-autoscaler approach while maintaining our latency targets. We use similar Temporal workflows for parts of the GPU host lifecycle, including pool adjustment and health management.

Overlapping input preparation with GPU inference#

GPU inference workers generally process one inference at a time. However, the beginning of each activity is often CPU- or network-bound: the worker downloads audio, reference footage, and preprocessed assets before it uses the GPU.

With strict single-activity concurrency, the GPU remains idle during every input download:

Serial:

[ download ][ inference ][ upload ][ download ][ inference ]
  GPU idle                 GPU idle

We wanted to begin preparing the next task shortly before the current inference finished:

Overlapped:

[ download ][ inference ][ upload ]
              [ download ][ inference ]

Simply setting activity concurrency to two does not provide the behavior we want. The second activity would begin immediately and then wait inside its execution until the GPU became available.

That waiting time would be recorded as part of the activity's start_to_close duration. We use that metric for performance monitoring and cost analysis, so allowing activities to spend significant time parked inside their execution would make it less representative of actual work.

Instead, we implemented a custom Temporal SlotSupplier. A slot supplier controls when a worker may accept another activity.

Under normal operation, the supplier exposes one base slot. Shortly before the current GPU inference is expected to finish, the activity signals that the worker may admit one additional task for input preparation:

class PipelineSlotSupplier(CustomSlotSupplier):
    """One base slot plus a signal-controlled overlap slot."""

    def _admittable(self) -> bool:
        # Keep the base slot available so an idle worker can poll.
        return self._active == 0 or (
            self._active < self._max
            and self._credits > 0
        )

    def signal_ready(self) -> None:
        """Allow one additional task to begin preparing its inputs."""
        with self._lock:
            if self._credits >= 1:
                return

            self._credits += 1

            self._wake_pending_reservation()

The signal creates one admission credit. Temporal can then deliver the next activity, which begins downloading its inputs while the current inference is finishing. A process-wide semaphore remains the authoritative guarantee that only one activity can use the GPU at a time.

Several safeguards are built into the implementation:

• The base slot remains available. A worker needs a slot to poll for work, so gating every slot could leave an idle worker unable to receive a task.

• Admission credits are capped at one and are idempotent. Multiple signals cannot create an unbounded set of prefetched activities.

• Credits are cleared when the worker becomes idle. This prevents an old signal from admitting a task too early at a later time.

• The autoscaler still counts the worker as one GPU unit. Admitting an additional preparation task does not create additional inference capacity.

• The GPU semaphore remains authoritative. Regardless of activity admission, only one activity can use the GPU at a time.

The failure mode is intentionally simple. If the overlap signal is not sent, the worker returns to serial execution. Throughput may be lower, but the worker does not double-book the GPU or leave an activity waiting indefinitely.

What Temporal changed for us#

Temporal did not remove the complexity of AI video generation—it changed the kind of complexity we have to manage.

We no longer spend engineering effort keeping schedulers, brokers, databases, and callbacks consistent after failures. Instead, we focus on workflow logic, GPU utilization, and product behavior, while Temporal provides durable execution, recovery, and execution history.

Today, that execution model coordinates millions of workflow executions and tens of millions of activity executions each day across roughly 300 workflow types, nearly 1,000 activities, around 100 task queues, and more than 50 independently scaled worker deployments spanning multiple GPU providers, and more importantly, it has increased our time to ship complex workflows. As the platform has grown, the execution model has remained consistent, making workflows easier to operate, extend, and recover than the custom orchestration system it replaced.