| This is a guest post from Austin Bennett, Director of Engineering at Azra AI |
When engineering teams first adopt Temporal, it’s usually for asynchronous, long-running processes: multi-day human-in-the-loop workflows, complex batch pipelines, or distributed sagas. In those domains, Temporal’s value is obvious.
At Azra AI, our journey began somewhere entirely different: in the synchronous hot path of our core clinical API.
This Workflow was our team’s very first introduction of Temporal into our engineering stack. Today, Azra has multiple dedicated engineering groups built around using Temporal for a wide array of mission-critical purposes—from our multi-tenant activity automation engine and diagnostic laboratory pipelines to platform-wide observability and client-side HIPAA encryption. But our entire Temporal journey began right here: in our core clinical API, solving an immediate, high-stakes distributed consistency problem in our patient-insights service.
In our world, an oncology nurse or patient coordinator confirms a clinical activity or updates a patient record in our web UI. That single interaction carries two requirements that are difficult to meet together:
- The user needs an instantaneous synchronous response: The client must receive HTTP 200 with the newly minted entity ID in sub-second time to update the UI and proceed with clinical workflows.
- The healthcare system requires 100% guaranteed durability: That mutation must trigger immutable audit entries for HIPAA compliance, update downstream care team queues, sync with clinical analytics, and trigger external registries. If any of these side effects fail, patient safety and regulatory compliance are at risk.
To solve this, we standardized on the sync-head / async-tail architectural pattern, built directly on Temporal’s modern UpdateWithStart primitive.
In this post, we’ll explore why traditional solutions (dual-writes, outboxes, and 202-polling) break down, dissect the internals of UpdateWithStart, and show how we implemented this pattern in production using the Temporal Java/Kotlin SDK.
The distributed systems dilemma: why traditional solutions fall short#
Consider a typical endpoint: POST /api/v2/activities. The API must write to a primary database and publish an audit/analytics event.
POST /api/v2/activities
│
┌─────────────┴─────────────┐
▼ ▼
┌──────────────────┐ ┌───────────────────┐
│ Write to DB │ │ Publish Events │
│ (Sub-second) │ │ & Audit Trail │
│ Returns ID │ │ (Must NOT drop) │
└──────────────────┘ └───────────────────┘
Historically, teams attempt to solve this in one of four ways:
| Approach | Mechanics | Why It Fails in Mission-Critical Systems |
|---|---|---|
| All-Synchronous Execution | The HTTP thread calls the database, calls Kafka, and calls third-party APIs before returning. | Cascading latency and failure. If Kafka is rebalancing or a downstream analytics service degrades, the user’s UI times out or returns HTTP 500—even though the core database write might have succeeded. |
| The Dual-Write Anti-Pattern | Commit to the database, then fire an asynchronous event to a message broker. | Non-atomic by definition. If the application pod crashes, restarts, or loses network connectivity between the DB commit and the broker ack, the message is permanently lost. |
| Transactional Outbox (CDC) | Write the entity and an event record in the same DB transaction; tail the database WAL via Debezium or poll an outbox table. |
High operational overhead. Outbox tables add database bloat, CDC schema evolution challenges, polling lag, and provide no native support for multi-step compensating actions or complex retries. |
| Pure Async Workflow (202 Accepted) | Return 202 Accepted immediately; let Temporal execute the entire pipeline asynchronously. |
Painful client ergonomics. Simple CRUD operations now require polling loops, WebSocket subscriptions, or fragile optimistic UI states just to obtain a generated database ID. |
We didn’t want to trade developer experience for reliability, or latency for consistency. We wanted:
- Sub-second API latency for the caller.
- Guaranteed durable execution for all downstream side effects.
- Zero dual-write inconsistency by letting a single orchestrator manage both phases.
The concept: sync-head / async-tail#
The sync-head / async-tail pattern separates an operation into two distinct phases managed within a single, unified Temporal Workflow:
Figure 1: The sync-head / async-tail sequence diagram
Phase 1: the sync head (fail-fast)#
The “Head” represents the minimal critical path required to persist state and generate an identity.
- It executes synchronously while the client’s HTTP socket remains open.
- It uses strict, fail-fast retry budgets (e.g., max 3 attempts, 1-second initial interval, short timeouts). If the primary database is deadlocked or unreachable, we want to know immediately and return HTTP 500 to the caller.
Phase 2: the async tail (durable backoff)#
The “Tail” represents all subsequent side effects: audit logs, downstream notifications, and analytics streaming.
- The tail begins after the database write completes. The API waits only for the head; it does not wait for the downstream work to finish.
- It uses broad, resilient retry policies (e.g., exponential backoff capped at 60s, bounded by a 30-minute
ScheduleToCloseTimeoutwith no maximum attempt limit). If a downstream destination such as Kafka, another database, or a third-party webhook is unavailable, Temporal retries the Activity according to its configured retry policy. - The client is completely decoupled from downstream turbulence.
The missing primitive: why UpdateWithStart?#
Before Workflow Updates were introduced, external callers communicated with Temporal via Signals (asynchronous, fire-and-forget) and Queries (synchronous, read-only). To simulate synchronous request-response interactions, engineers historically resorted to clumsy workarounds:
SignalWithStartfollowed by a client-side polling loop querying the Workflow.- Starting a Workflow and having an Activity post the result back to Redis/Memcached where the API handler waited on a countdown latch. Both approaches introduced race conditions, latency overhead, and extra infrastructure.
Workflow Updates vs. Signals#
Temporal Updates changed the game. An Update is a first-class synchronous RPC that:
- Can accept parameters.
- Runs an optional validator (which executes before anything is written to the workflow history).
- Mutates Workflow state.
- Returns a strongly-typed result to the caller once a condition is met.
The Power of UpdateWithStart#
UpdateWithStart takes this a step further: In a single atomic round-trip, the client instructs Temporal: "Start this workflow if it is not already running, and simultaneously deliver this Update to it. Keep the connection open until the Update reaches the requested lifecycle stage."
Because the initiation and the update are atomic, there is zero chance of a race condition between starting the workflow and issuing the update. However, keep in mind that this atomicity applies to the server-side request; if the Update's business logic or validation later fails, the newly started Workflow will still remain running in the background.
Production implementation (Kotlin & Micronaut)#
Let’s look at the actual code structure powering this pattern in Azra’s patient-insights service.
-
The Workflow contract
We define our Workflow with both a
@WorkflowMethod(for the overall lifecycle) and an@UpdateMethodpaired with an@UpdateValidatorMethod:
@WorkflowInterface
interface ActivityWorkflow {
/**
* Workflow entry point: executes the full pipeline
* (DB Write Head -> Analytics & Audit Tail).
*/
@WorkflowMethod
fun execute(input: ActivityWorkflowInput)
/**
* Validates the update request prior to recording in History.
*/
@UpdateValidatorMethod(updateName = "awaitDbWrite")
fun validateAwaitDbWrite()
/**
* Synchronous Update endpoint.
* Blocks until the database write completes, then returns the result.
*/
@UpdateMethod
fun awaitDbWrite(): ActivityWorkflowResult
}
2. The Workflow state machine#
Inside the implementation, the Workflow orchestrates two Activity stubs with radically different reliability profiles, using Workflow.await to bridge the synchronous update:
class ActivityWorkflowImpl : ActivityWorkflow {
// Internal workflow state capturing the Head's output
private var dbResult: ActivityWorkflowResult? = null
// Activity 1 (HEAD): Fail-fast retry — the HTTP caller is waiting!
private val dbActivity = Workflow.newActivityStub(
ActivityDbActivity::class.java,
ActivityOptions.newBuilder()
.setStartToCloseTimeout(Duration.ofSeconds(10))
.setScheduleToCloseTimeout(Duration.ofSeconds(20))
.setHeartbeatTimeout(Duration.ofSeconds(5))
.setRetryOptions(
RetryOptions.newBuilder()
.setInitialInterval(Duration.ofSeconds(1))
.setMaximumAttempts(3)
.build()
)
.build()
)
// Activity 2 (TAIL): Resilient backoff — runs in the background
private val analyticsActivity = Workflow.newActivityStub(
ActivityAnalyticsActivity::class.java,
ActivityOptions.newBuilder()
.setStartToCloseTimeout(Duration.ofSeconds(30))
.setScheduleToCloseTimeout(Duration.ofMinutes(30))
.setRetryOptions(
RetryOptions.newBuilder()
.setInitialInterval(Duration.ofSeconds(1))
.setBackoffCoefficient(2.0)
.setMaximumInterval(Duration.ofSeconds(60))
// No maximumAttempts: retries until ScheduleToCloseTimeout (30m)
.build()
)
.build()
)
override fun execute(input: ActivityWorkflowInput) {
// Step 1: Execute DB write (The Sync Head)
// Storing dbResult immediately satisfies awaitDbWrite()
dbResult = dbActivity.writeToDb(input)
// Step 2: Execute Analytics & Audit Publishing (The Async Tail)
// The API caller does not wait for this Activity to complete.
val result = checkNotNull(dbResult) { "DB result must not be null" }
analyticsActivity.buildAndPublishAuditMessage(input, result)
}
override fun validateAwaitDbWrite() {
// Validation logic can inspect current state or input
}
override fun awaitDbWrite(): ActivityWorkflowResult {
// Block the update caller until dbResult is populated by Step 1
val completed = Workflow.await(Duration.ofSeconds(30)) { dbResult != null }
if (!completed) {
throw ApplicationFailure.newFailure(
"DB write did not complete within 30-second budget",
"TIMEOUT"
)
}
return checkNotNull(dbResult)
}
}
How the state machine unblocks#
- When
startUpdateWithStartis received by the cluster, the Workflow begins and the Update registers a listener ondbResult. - The Worker picks up the Workflow Task and dispatches
dbActivity.writeToDb(input). - When the Activity finishes, the worker starts the next Workflow Task. It assigns
dbResult = .... - In the same Workflow Task, the condition in
Workflow.await { dbResult != null }evaluates totrue. - The
awaitDbWriteupdate completes, returningActivityWorkflowResultover the gRPC stream to the client. - The Workflow Task continues to the next statement: dispatching
analyticsActivity.
What if a downstream outage outlasts the retry policy?#
Because analyticsActivity omits maximumAttempts and bounds retries by ScheduleToCloseTimeout (30 minutes) with exponential backoff capped at 60 seconds, Temporal automatically rides out transient broker or webhook outages (such as a 10-minute maintenance window or cluster rebalance) without holding open the caller’s HTTP connection. If a prolonged outage exceeds that 30-minute window and exhausts the retry budget, the synchronous database write is never rolled back—the caller has already received its HTTP response and the database state remains valid. Instead of silently dropping the event like an in-memory fire-and-forget thread, the
Workflow Execution transitions to Failed in Temporal, firing platform alerts. Because the head Activity stamps the workflowId as an idempotency key on the database row within the same transaction and records the serialized audit payload in Event History, operators can safely Reset the failed Workflow Execution once the downstream system recovers: the head Activity detects the existing idempotency key, skips the duplicate DB write, and the tail Activity publishes the audit event.
3. The API starter#
In the API service layer, we invoke startUpdateWithStart:
@Singleton
class ActivityWorkflowStarter(
private val config: TemporalConfig,
private val clientProvider: TemporalClientProvider
) {
private val log = LoggerFactory.getLogger(ActivityWorkflowStarter::class.java)
@WithSpan("ActivityWorkflow.executeAndWaitForDbWrite")
fun executeAndWaitForDbWrite(input: ActivityWorkflowInput): ActivityWorkflowResult {
val taskQueue = config.resolveActivityCrudTaskQueue()
val workflowId = "${config.clientId}:${input.workflowId}"
val options = WorkflowOptions.newBuilder()
.setTaskQueue(taskQueue)
.setWorkflowId(workflowId)
.setWorkflowIdConflictPolicy(
WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING
)
.build()
val workflow = clientProvider.workflowClient.newWorkflowStub(
ActivityWorkflow::class.java,
options
)
// 1. Define the Start operation (points to execute)
val startOp = WithStartWorkflowOperation<Void, ActivityWorkflowInput>(
workflow::execute,
input
)
// 2. Start the Workflow if needed and send the awaitDbWrite
val updateHandle = WorkflowClient.startUpdateWithStart<ActivityWorkflowResult>(
workflow::awaitDbWrite,
UpdateOptions.newBuilder(ActivityWorkflowResult::class.java)
// Crucial: Wait for COMPLETED stage so we receive the return value
.setWaitForStage(WorkflowUpdateStage.COMPLETED)
.build(),
startOp
)
// 3. Blocks ONLY until Activity 1 completes!
val result = updateHandle.result
log.info(
"Sync Head completed: workflowId={} entityId={}",
workflowId,
result.activityId
)
return result
}
}
Notice UpdateOptions.newBuilder().setWaitForStage(WorkflowUpdateStage.COMPLETED). Temporal Updates support two stages:
ACCEPTED: Unblocks as soon as the validator passes and the Update is committed to Event History.-
COMPLETED: Unblocks when the update function actually returns a value. For sync-head / async-tail,WorkflowUpdateStage.COMPLETEDis mandatory because we are waiting for the head Activity to produce our return payload.
Production lessons & architectural trade-offs#
Deploying this pattern at scale in healthcare infrastructure taught us several critical lessons:
1. The fallback fallacy: choose hard-fail semantics#
When introducing Temporal into synchronous request paths, engineers frequently suggest: “If Temporal is down, can we fall back to writing directly to PostgreSQL and pushing to PubSub?”
We explicitly chose hard-fail semantics. If Temporal is unreachable, the API returns HTTP 500.
- Dual code paths guarantee bugs: A fallback path that only runs during incidents is inherently untested and prone to catastrophic failure.
- Preventing data drift: If the fallback writes to the database but fails to publish downstream events, you recreate the very dual-write bugs you introduced Temporal to eliminate.
- Availability: Backed by Temporal Cloud or multi-zone GKE clusters with automated failover, Temporal’s availability matches or exceeds our relational databases.
2. Idempotency via conflict policies#
What happens if an oncology coordinator clicks “Submit” twice, or a mobile client retries an HTTP request? By scoping the workflowId to the business operation (e.g., client-104:activity-create-<clientMutationId>), we set:
.setWorkflowIdConflictPolicy( WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING
)
If a duplicate request arrives while the Workflow is already running, Temporal reuses the existing Workflow Execution and attaches the Update, preventing duplicate database writes without requiring distributed locks.
3. Workflow History hygiene#
Workflows implementing this pattern should stay lean. Because the Workflow only coordinates 2–3 Activities, the Event History typically contains fewer than 20 events. This ensures that History replay is instantaneous and Worker memory overhead remains negligible.
4. Distributed tracing across the sync/async split#
In our API starter, we annotate the method with OpenTelemetry:
@WithSpan("ActivityWorkflow.executeAndWaitForDbWrite")
Because the HTTP request terminates before the workflow completes, end-to-end tracing is vital. The Temporal Java SDK transparently propagates traceparent headers into Workflow and Activity headers, allowing our SRE team to trace from the initial HTTP POST all the way to the final asynchronous Kafka publish in Jaeger / Google Cloud Trace.
5. The adoption catalyst: from one endpoint to dedicated engineering groups#
Choosing a latency-sensitive, synchronous-to-asynchronous boundary as our first Temporal implementation was a pivotal decision that shaped how Azra builds software today:
- Debunking the “slow orchestrator” myth early: A frequent hesitation when evaluating workflow engines is the fear of latency overhead in user-facing paths. In our GKE (
us-central1) environment calling Temporal Cloud and Cloud SQL (MySQL 8.4), our synchronous head (executeAndWaitForDbWrite) completes with a median (p50) latency of 268 ms andp95of 370 ms (p99of 454 ms, min 214 ms across n=229n = 229n=229 recordedCREATE,UPDATE, andDELETEexecutions). Breaking down that median latency, roughly 67 ms is spent on the initialUpdateWithStartgRPC round-trip to Temporal Cloud and Worker dispatch, and 200 ms is spent executing the MySQL transaction, serializing the protobuf audit snapshot, and streaming the completed Update result back to the API pod. Meanwhile, the asynchronous Pub/Sub tail (p50of 71 ms,p95of 443 ms, max 1,152 ms) runs entirely after the HTTP caller unblocks—keeping API response times comfortably sub-second even when downstream messaging spikes over a second. Proving this with real telemetry dispelled any skepticism across the engineering organization about running Temporal in the hot path. - The adoption wedge: The immediate reliability wins and zero-data-loss guarantees of this first Workflow created strong internal champions. Developers saw that they no longer had to maintain fragile outbox tables, CDC jobs, or manual retry loops.
- Forming dedicated Temporal groups: That single initial Workflow in
patient-insightsbecame the operational blueprint for our entire platform. Today, Azra has multiple dedicated engineering groups built around Temporal:- Activity Automation Engine (AAE): A dedicated multi-tenant Go service orchestrating patient activity generation, care team queue assignments, and due-date calculations across independent tenant queues.
- Diagnostic laboratory integrations: Event-driven sagas ingesting external diagnostic webhooks (e.g., Redox/Biodesix lung cancer assays), performing patient matching, scanning and storing PDFs, and driving oncology patient care paths.
- Clinical Trial Management System (CTMS): A dedicated intelligence ecosystem, where Python-based Temporal Workflows orchestrate high-throughput clinical document ingestion, tenant-sharded OCR sweeps, and AI-driven biomarker extraction. Temporal coordinates the evaluation of inclusion/exclusion criteria against oncology patient records, matching eligible cancer patients to life-saving clinical trial protocols in real time while enforcing tenant-lane isolation.
- Enterprise platform & security: Platform Engineering managing cluster infrastructure, custom Prometheus alerting rules for non-determinism, and client-side AES-256-GCM payload encryption with Cloud KMS for HIPAA compliance.
What began as a targeted architectural solution for a single thorny API endpoint grew into the operational backbone of our distributed architecture.
Conclusion#
The sync-head / async-tail pattern bridges the gap between synchronous user expectations and distributed, durable guarantees.
By using Temporal’s UpdateWithStart:
- We eliminated the dual-write problem without managing complex Transactional Outbox CDC infrastructure.
- Our API waits for the head to finish, without waiting for downstream publishing.
- Downstream side effects are bulletproof, benefiting from Temporal’s durable execution, automated retries, and operational visibility.
If your team is building APIs that orchestrate mission-critical mutations, you don’t have to choose between fast endpoints and durable guarantees. With UpdateWithStart, you can have both.
