---
title: "The sync-head / async-tail pattern: fast APIs with durable execution using  Temporal Update-With-Start"
canonical: "https://temporal.io/blog/fast-apis-with-durable-execution-using-temporal-update-with-start.md"
source: "https://temporal.io/blog/fast-apis-with-durable-execution-using-temporal-update-with-start"
published: "2026-11-17"
authors:
  - "Austin Bennett"
category: "Community"
tags:
  - "AI/ML"
  - "Code Samples"
reading_time_minutes: 14
---

# The sync-head / async-tail pattern: fast APIs with durable execution using Temporal Update-With-Start

|        |
| ---------- |
| *This is a guest post from Austin Bennett, Director of Engineering at Azra AI*           |
|        |

When engineering teams first adopt Temporal, it’s usually for asynchronous, long-running processes: multi-day human-in-the-loop workflows, complex batch pipelines, or distributed sagas. In those domains, Temporal’s value is obvious.

At [Azra AI](https://www.azra-ai.com/), our journey began somewhere entirely different: __in the synchronous hot path of our core clinical API__.

This Workflow was our team’s very first introduction of Temporal into our engineering stack. Today, Azra has multiple dedicated engineering groups built around using Temporal for a wide array of mission-critical purposes—from our multi-tenant activity automation engine and diagnostic laboratory pipelines to platform-wide observability and client-side HIPAA encryption. But our entire Temporal journey began right here: in our core clinical API, solving an immediate, high-stakes distributed consistency problem in our `patient-insights` service.

In our world, an oncology nurse or patient coordinator confirms a clinical activity or updates a patient record in our web UI. That single interaction carries two requirements that are difficult to meet together:

1. __The user needs an instantaneous synchronous response__: The client must receive HTTP 200 with the newly minted entity ID in sub-second time to update the UI and proceed with clinical workflows.
2. __The healthcare system requires 100% guaranteed durability__: That mutation must trigger immutable audit entries for HIPAA compliance, update downstream care team queues, sync with clinical analytics, and trigger external registries. If any of these side effects fail, patient safety and regulatory compliance are at risk.

To solve this, we standardized on the __sync-head / async-tail__ architectural pattern, built directly on Temporal’s modern `UpdateWithStart` primitive.

In this post, we’ll explore why traditional solutions (dual-writes, outboxes, and 202-polling) break down, dissect the internals of `UpdateWithStart`, and show how we implemented this pattern in production using the Temporal Java/Kotlin SDK.

## The distributed systems dilemma: why traditional solutions fall short
Consider a typical endpoint: POST `/api/v2/activities`. The API must write to a primary database and publish an audit/analytics event.

                 POST /api/v2/activities
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
      ┌──────────────────┐       ┌───────────────────┐
      │   Write to DB    │       │  Publish Events   │
      │   (Sub-second)   │       │  & Audit Trail    │
      │   Returns ID     │       │  (Must NOT drop)  │
      └──────────────────┘       └───────────────────┘

Historically, teams attempt to solve this in one of four ways:

| Approach    | Mechanics     |  Why It Fails in Mission-Critical Systems    |
| ---------- | ---------- | ---------- |
| __All-Synchronous Execution__       | The HTTP thread calls the database, calls Kafka, and calls third-party APIs before returning.       | __Cascading latency and failure__. If Kafka is rebalancing or a downstream analytics service degrades, the user’s UI times out or returns HTTP 500—even though the core database write might have succeeded.       |
| __The Dual-Write Anti-Pattern__       | Commit to the database, then fire an asynchronous event to a message broker.       | __Non-atomic by definition__. If the application pod crashes, restarts, or loses network connectivity between the DB commit and the broker ack, the message is permanently lost.       |
| __Transactional Outbox (CDC)__       | Write the entity and an event record in the same DB transaction; tail the database WAL via Debezium or poll an `outbox` table.       | __High operational overhead__. Outbox tables add database bloat, CDC schema evolution challenges, polling lag, and provide no native support for multi-step compensating actions or complex retries.       |
| __Pure Async Workflow (202 Accepted)__       | Return `202 Accepted` immediately; let Temporal execute the entire pipeline asynchronously.      | __Painful client ergonomics__. Simple CRUD operations now require polling loops, WebSocket subscriptions, or fragile optimistic UI states just to obtain a generated database ID.       |

We didn’t want to trade developer experience for reliability, or latency for consistency. We wanted: 
- __Sub-second API latency__ for the caller. 
- __Guaranteed durable execution__ for all downstream side effects. 
- __Zero dual-write inconsistency__ by letting a single orchestrator manage both phases.

## The concept: sync-head / async-tail
The __sync-head / async-tail__ pattern separates an operation into two distinct phases managed within a single, unified Temporal Workflow:

![Figure 1: The sync-head / async-tail sequence diagram](https://images.ctfassets.net/0uuz8ydxyd9p/1gHdcKKdc9OlCdrzGyvZi6/8f3ce1db510b734cb659f843b5f5402b/image1.png){.zoom}

<p style="text-align: center;"><em>Figure 1: The sync-head / async-tail sequence diagram</em></p>

### Phase 1: the sync head (fail-fast)
The “Head” represents the minimal critical path required to persist state and generate an identity.
- It executes synchronously while the client’s HTTP socket remains open. 
- It uses __strict, fail-fast retry budgets__ (e.g., max 3 attempts, 1-second initial interval, short timeouts). If the primary database is deadlocked or unreachable, we want to know immediately and return HTTP 500 to the caller.

### Phase 2: the async tail (durable backoff)
The “Tail” represents all subsequent side effects: audit logs, downstream notifications, and analytics streaming. 
- The tail begins after the database write completes. The API waits only for the head; it does not wait for the downstream work to finish. 
- It uses __broad, resilient retry policies__ (e.g., exponential backoff capped at 60s, bounded by a 30-minute `ScheduleToCloseTimeout` with no maximum attempt limit). If a downstream destination such as Kafka, another database, or a third-party webhook is unavailable, Temporal retries the Activity according to its configured retry policy.
- The client is completely decoupled from downstream turbulence.

## The missing primitive: why `UpdateWithStart`?
Before Workflow Updates were introduced, external callers communicated with Temporal via __Signals__ (asynchronous, fire-and-forget) and __Queries__ (synchronous, read-only).
To simulate synchronous request-response interactions, engineers historically resorted to clumsy workarounds: 
1. `SignalWithStart` followed by a client-side polling loop querying the Workflow. 
2. Starting a Workflow and having an Activity post the result back to Redis/Memcached where the API handler waited on a countdown latch.
Both approaches introduced race conditions, latency overhead, and extra infrastructure.

### Workflow Updates vs. Signals
Temporal __Updates__ changed the game. An Update is a first-class synchronous RPC that: 
- Can accept parameters. 
- Runs an optional __validator__ (which executes before anything is written to the workflow history). 
- Mutates Workflow state. 
- Returns a strongly-typed result to the caller once a condition is met.

### The Power of `UpdateWithStart`
`UpdateWithStart` takes this a step further: In a single atomic round-trip, the client instructs Temporal: "Start this workflow if it is not already running, and simultaneously deliver this Update to it. Keep the connection open until the Update reaches the requested lifecycle stage." 

Because the initiation and the update are atomic, there is zero chance of a race condition between starting the workflow and issuing the update. However, keep in mind that this atomicity applies to the server-side request; if the Update's business logic or validation later fails, the newly started Workflow will still remain running in the background.

## Production implementation (Kotlin & Micronaut)
Let’s look at the actual code structure powering this pattern in Azra’s `patient-insights` service.
1. The Workflow contract
We define our Workflow with both a `@WorkflowMethod` (for the overall lifecycle) and an `@UpdateMethod` paired with an `@UpdateValidatorMethod`:

```kotlin
@WorkflowInterface
interface ActivityWorkflow {

    /**
     * Workflow entry point: executes the full pipeline
     * (DB Write Head -> Analytics & Audit Tail).
     */
    @WorkflowMethod
    fun execute(input: ActivityWorkflowInput)

    /**
     * Validates the update request prior to recording in History.
     */
    @UpdateValidatorMethod(updateName = "awaitDbWrite")
    fun validateAwaitDbWrite()

    /**
     * Synchronous Update endpoint.
     * Blocks until the database write completes, then returns the result.
     */
    @UpdateMethod
    fun awaitDbWrite(): ActivityWorkflowResult
}
```

### 2. The Workflow state machine
Inside the implementation, the Workflow orchestrates two Activity stubs with radically different reliability profiles, using `Workflow.await` to bridge the synchronous update:

```kotlin
class ActivityWorkflowImpl : ActivityWorkflow {

    // Internal workflow state capturing the Head's output
    private var dbResult: ActivityWorkflowResult? = null

    // Activity 1 (HEAD): Fail-fast retry — the HTTP caller is waiting!
    private val dbActivity = Workflow.newActivityStub(
        ActivityDbActivity::class.java,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(10))
            .setScheduleToCloseTimeout(Duration.ofSeconds(20))
            .setHeartbeatTimeout(Duration.ofSeconds(5))
            .setRetryOptions(
                RetryOptions.newBuilder()
                    .setInitialInterval(Duration.ofSeconds(1))
                    .setMaximumAttempts(3)
                    .build()
            )
            .build()
    )

   // Activity 2 (TAIL): Resilient backoff — runs in the background
   private val analyticsActivity = Workflow.newActivityStub(
       ActivityAnalyticsActivity::class.java,
       ActivityOptions.newBuilder()
           .setStartToCloseTimeout(Duration.ofSeconds(30))
           .setScheduleToCloseTimeout(Duration.ofMinutes(30))
           .setRetryOptions(
               RetryOptions.newBuilder()
                   .setInitialInterval(Duration.ofSeconds(1))
                   .setBackoffCoefficient(2.0)
                   .setMaximumInterval(Duration.ofSeconds(60))
                   // No maximumAttempts: retries until ScheduleToCloseTimeout (30m)
                   .build()
           )
           .build()
   )

    override fun execute(input: ActivityWorkflowInput) {
        // Step 1: Execute DB write (The Sync Head)
        // Storing dbResult immediately satisfies awaitDbWrite()
        dbResult = dbActivity.writeToDb(input)

        // Step 2: Execute Analytics & Audit Publishing (The Async Tail)
        // The API caller does not wait for this Activity to complete. 
        val result = checkNotNull(dbResult) { "DB result must not be null" }
        analyticsActivity.buildAndPublishAuditMessage(input, result)
    }

    override fun validateAwaitDbWrite() {
        // Validation logic can inspect current state or input
    }

    override fun awaitDbWrite(): ActivityWorkflowResult {
        // Block the update caller until dbResult is populated by Step 1
        val completed = Workflow.await(Duration.ofSeconds(30)) { dbResult != null }
        if (!completed) {
            throw ApplicationFailure.newFailure(
                "DB write did not complete within 30-second budget", 
                "TIMEOUT"
            )
        }
        return checkNotNull(dbResult)
    }
}
```

#### How the state machine unblocks
1. When `startUpdateWithStart` is received by the cluster, the Workflow begins and the Update registers a listener on `dbResult`.
2. The Worker picks up the Workflow Task and dispatches `dbActivity.writeToDb(input)`.
3. When the Activity finishes, the worker starts the next Workflow Task. It assigns `dbResult = ...`.
4. In the same Workflow Task, the condition in `Workflow.await { dbResult != null }` evaluates to `true`.
5. The `awaitDbWrite` update completes, returning `ActivityWorkflowResult` over the gRPC stream to the client.
6. The Workflow Task continues to the next statement: dispatching `analyticsActivity`.

#### What if a downstream outage outlasts the retry policy? 
Because `analyticsActivity` omits `maximumAttempts` and bounds retries by `ScheduleToCloseTimeout` (30 minutes) with exponential backoff capped at 60 seconds, Temporal automatically rides out transient broker or webhook outages (such as a 10-minute maintenance window or cluster rebalance) without holding open the caller’s HTTP connection. If a prolonged outage exceeds that 30-minute window and exhausts the retry budget, the synchronous database write is never rolled back—the caller has already received its HTTP response and the database state remains valid. Instead of silently dropping the event like an in-memory fire-and-forget thread, the 

Workflow Execution transitions to `Failed` in Temporal, firing platform alerts. Because the head Activity stamps the `workflowId` as an idempotency key on the database row within the same transaction and records the serialized audit payload in Event History, operators can safely __Reset__ the failed Workflow Execution once the downstream system recovers: the head Activity detects the existing idempotency key, skips the duplicate DB write, and the tail Activity publishes the audit event.

### 3. The API starter
In the API service layer, we invoke `startUpdateWithStart`:

```kotlin
@Singleton
class ActivityWorkflowStarter(
    private val config: TemporalConfig,
    private val clientProvider: TemporalClientProvider
) {
    private val log = LoggerFactory.getLogger(ActivityWorkflowStarter::class.java)

    @WithSpan("ActivityWorkflow.executeAndWaitForDbWrite")
    fun executeAndWaitForDbWrite(input: ActivityWorkflowInput): ActivityWorkflowResult {
        val taskQueue = config.resolveActivityCrudTaskQueue()
        val workflowId = "${config.clientId}:${input.workflowId}"

       val options = WorkflowOptions.newBuilder()
           .setTaskQueue(taskQueue)
           .setWorkflowId(workflowId)
           .setWorkflowIdConflictPolicy(
               WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING
           )
           .build()

        val workflow = clientProvider.workflowClient.newWorkflowStub(
            ActivityWorkflow::class.java,
            options
        )

        // 1. Define the Start operation (points to execute)
        val startOp = WithStartWorkflowOperation<Void, ActivityWorkflowInput>(
            workflow::execute, 
            input
        )

        // 2. Start the Workflow if needed and send the awaitDbWrite
        val updateHandle = WorkflowClient.startUpdateWithStart<ActivityWorkflowResult>(
            workflow::awaitDbWrite,
            UpdateOptions.newBuilder(ActivityWorkflowResult::class.java)
                // Crucial: Wait for COMPLETED stage so we receive the return value
                .setWaitForStage(WorkflowUpdateStage.COMPLETED)
                .build(),
            startOp
        )

        // 3. Blocks ONLY until Activity 1 completes!
        val result = updateHandle.result

        log.info(
            "Sync Head completed: workflowId={} entityId={}", 
            workflowId, 
            result.activityId
        )

        return result
    }
}
```

Notice `UpdateOptions.newBuilder().setWaitForStage(WorkflowUpdateStage.COMPLETED)`. Temporal Updates support two stages: 
- `ACCEPTED`: Unblocks as soon as the validator passes and the Update is committed to Event History. 
- `COMPLETED`: Unblocks when the update function actually returns a value.
For sync-head / async-tail, `WorkflowUpdateStage.COMPLETED` is mandatory because we are waiting for the head Activity to produce our return payload.

## Production lessons & architectural trade-offs
Deploying this pattern at scale in healthcare infrastructure taught us several critical lessons:

### 1. The fallback fallacy: choose hard-fail semantics
When introducing Temporal into synchronous request paths, engineers frequently suggest: *“If Temporal is down, can we fall back to writing directly to PostgreSQL and pushing to PubSub?”*

__We explicitly chose hard-fail semantics__. If Temporal is unreachable, the API returns HTTP 500. 
- __Dual code paths guarantee bugs__: A fallback path that only runs during incidents is inherently untested and prone to catastrophic failure. 
- __Preventing data drift__: If the fallback writes to the database but fails to publish downstream events, you recreate the very dual-write bugs you introduced Temporal to eliminate. 
- __Availability__: Backed by Temporal Cloud or multi-zone GKE clusters with automated failover, Temporal’s availability matches or exceeds our relational databases.

### 2. Idempotency via conflict policies
What happens if an oncology coordinator clicks “Submit” twice, or a mobile client retries an HTTP request? By scoping the `workflowId` to the business operation (e.g., `client-104:activity-create-<clientMutationId>`), we set:
`.setWorkflowIdConflictPolicy(
    WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING`
)
If a duplicate request arrives while the Workflow is already running, Temporal reuses the existing Workflow Execution and attaches the Update, preventing duplicate database writes without requiring distributed locks.

### 3. Workflow History hygiene
Workflows implementing this pattern should stay lean. Because the Workflow only coordinates 2–3 Activities, the Event History typically contains fewer than 20 events. This ensures that History replay is instantaneous and Worker memory overhead remains negligible.

### 4. Distributed tracing across the sync/async split
In our API starter, we annotate the method with OpenTelemetry:
`@WithSpan("ActivityWorkflow.executeAndWaitForDbWrite")`
Because the HTTP request terminates before the workflow completes, end-to-end tracing is vital. The Temporal Java SDK transparently propagates traceparent headers into Workflow and Activity headers, allowing our SRE team to trace from the initial HTTP POST all the way to the final asynchronous Kafka publish in Jaeger / Google Cloud Trace.

### 5. The adoption catalyst: from one endpoint to dedicated engineering groups
Choosing a latency-sensitive, synchronous-to-asynchronous boundary as our *first* Temporal implementation was a pivotal decision that shaped how Azra builds software today:
- __Debunking the “slow orchestrator” myth early__: A frequent hesitation when evaluating workflow engines is the fear of latency overhead in user-facing paths. In our GKE (`us-central1`) environment calling Temporal Cloud and Cloud SQL (MySQL 8.4), our synchronous head (`executeAndWaitForDbWrite`) completes with a __median__ (`p50`) __latency of 268 ms__ and `p95` __of 370 ms__ (`p99` of 454 ms, min 214 ms across n=229n = 229n=229 recorded `CREATE`, `UPDATE`, and `DELETE` executions). Breaking down that median latency, roughly __67 ms__ is spent on the initial `UpdateWithStart` gRPC round-trip to Temporal Cloud and Worker dispatch, and __200 ms__ is spent executing the MySQL transaction, serializing the protobuf audit snapshot, and streaming the completed Update result back to the API pod. Meanwhile, the asynchronous Pub/Sub tail (`p50` of 71 ms, `p95` of 443 ms, max 1,152 ms) runs entirely after the HTTP caller unblocks—keeping API response times comfortably sub-second even when downstream messaging spikes over a second. Proving this with real telemetry dispelled any skepticism across the engineering organization about running Temporal in the hot path.
- __The adoption wedge__: The immediate reliability wins and zero-data-loss guarantees of this first Workflow created strong internal champions. Developers saw that they no longer had to maintain fragile outbox tables, CDC jobs, or manual retry loops.
- __Forming dedicated Temporal groups__: That single initial Workflow in `patient-insights` became the operational blueprint for our entire platform. Today, Azra has multiple dedicated engineering groups built around Temporal:
  - __Activity Automation Engine (AAE)__: A dedicated multi-tenant Go service orchestrating patient activity generation, care team queue assignments, and due-date calculations across independent tenant queues.
  - __Diagnostic laboratory integrations__: Event-driven sagas ingesting external diagnostic webhooks (e.g., Redox/Biodesix lung cancer assays), performing patient matching, scanning and storing PDFs, and driving oncology patient care paths.
  - __Clinical Trial Management System (CTMS)__: A dedicated intelligence ecosystem, where Python-based Temporal Workflows orchestrate high-throughput clinical document ingestion, tenant-sharded OCR sweeps, and AI-driven biomarker extraction. Temporal coordinates the evaluation of inclusion/exclusion criteria against oncology patient records, matching eligible cancer patients to life-saving clinical trial protocols in real time while enforcing tenant-lane isolation.
  - __Enterprise platform & security__: Platform Engineering managing cluster infrastructure, custom Prometheus alerting rules for non-determinism, and client-side AES-256-GCM payload encryption with Cloud KMS for HIPAA compliance.

What began as a targeted architectural solution for a single thorny API endpoint grew into the operational backbone of our distributed architecture.

## Conclusion
The __sync-head / async-tail__ pattern bridges the gap between synchronous user expectations and distributed, durable guarantees.

By using Temporal’s `UpdateWithStart`: 
- __We eliminated the dual-write problem__ without managing complex Transactional Outbox CDC infrastructure. 
- __Our API waits for the head to finish__, without waiting for downstream publishing. 
- __Downstream side effects are bulletproof__, benefiting from Temporal’s durable execution, automated retries, and operational visibility.

If your team is building APIs that orchestrate mission-critical mutations, you don’t have to choose between fast endpoints and durable guarantees. With `UpdateWithStart`, you can have both.
