Durable, flexible multi-agent systems

An agent system is a distributed system. You get to choose the framework and how much durability and human oversight the case demands; the tradeoffs are the part you don’t get to avoid.

For the last few months, I’ve been building one system to make that concrete: the same multi-agent fleet on Google ADK, on LangGraph, and on both at once, with Temporal as a layer underneath.

Where this started#

Ziggy’s is the playful imaginary ice cream shop I cooked up to show what it looks like when a multi-agent system handles fleet delivery around Las Vegas. We announced our ADK integration and wanted a demo for Google Cloud Next. The first version showed a multi-agent team assigning deliveries and dealing with an agent or driver disconnecting mid-route. For the AI Engineer World’s Fair, the fleet relocated to San Francisco, picked up LangGraph as a second framework, and got reframed around a harder version of “recover from a disconnect”: keeping a human in the loop. Disconnecting an agent is a machine failing. Waiting on a human is a machine succeeding at doing nothing, correctly, for as long as it takes. A human isn’t a function that returns in 200 milliseconds. They answer in minutes, hours, or after you’ve already redeployed twice.

The setup: Ziggy’s Ice Cream#

Here’s the operation. Orders come in from Moscone, Fisherman’s Wharf, and Chinatown; drivers batch pickups at the Ferry Building and deliver in sequence across downtown San Francisco.

Every order is handled by a small team of agents, not one model call:

• Fleet agent. Assesses the operational side: which drivers are available, where they are, and whether the fleet can take the order on.
• Customer agent. Assesses the order side: what’s being delivered, the destination, the timing, and anything about the customer that should shape the decision.
• Dispatch agent. Takes both assessments, makes the call, and assigns the delivery to a driver.

The Fleet and Customer agents run in parallel; Dispatch synthesizes their two views into one decision.

Three ways to run the fleet#

Quick grounding: Temporal runs your orchestration code as Workflows. Every step is journaled to an Event History, so a Workflow can die on one Worker and resume on another with nothing lost. Model and tool calls run as Activities, retryable steps in that same history. Everything here leans on that.

Under the jargon, an agent is a loop: observe, reason, act, repeat. The framework runs that loop; Temporal handles persistence, retries, and resumption.

The two frameworks think differently. Those differences matter because real systems often include multiple tools that solve similar problems.

You can watch the same team run three ways:

• All ADK. The Fleet, Customer, and Dispatch agents run on Google’s Agent Development Kit.
• All LangGraph. The same team is composed as a graph, looping from reason to act to evaluate.
• Cross-framework. Temporal orchestrates across the two: an ADK child Workflow runs the assessment, then hands its result to a LangGraph child Workflow that makes the dispatch decision. Two frameworks, one order, each with its own visible Event History.

Here’s that cross-framework handoff, with one child Workflow per framework:

# In the parent Workflow, one order runs across both frameworks:
# The ADK child assesses, then the LangGraph child dispatches from that assessment.

assessment = await workflow.execute_child_workflow(
    AdkAssessmentWorkflow.run,
    order,
    id=f"assess-{order_id}",
)

await workflow.start_child_workflow(
    LgDispatchWorkflow.run,
    LgDispatchInput(
        order,
        fleet_assessment=assessment.fleet_assessment,
        customer_assessment=assessment.customer_assessment,
    ),
    id=f"dispatch-{order_id}",
)

The same handoff, as a picture:

The two frameworks genuinely function differently. ADK leans agent-first: you compose teams, and it manages the conversation between them. LangGraph thinks in graphs: nodes and edges, with control flow explicit in the structure. Temporal plugs into both through an ADK plugin and the official LangGraph integration, recording each model and tool call as Activities in this demo.

So why split one order across both? It isn’t because one framework is categorically better at assessment and the other at dispatch, but the split isn’t arbitrary, either. It plays to each framework’s strengths: the assessment is a team of agents working in parallel, which is what ADK’s composition is for, while dispatch is a decision loop with an explicit branch to a human, which is what LangGraph’s graph and interrupt() are for.

In practice, the more common reason is that organizations rarely choose one framework cleanly. Different teams may prefer different tools, or services may have been built at different times; rewriting what already works is expensive. Cross-framework isn’t a party trick, and Temporal isn’t merely the glue between agent SDKs. When one team’s ADK service and another’s LangGraph service have to cooperate on the same order, Temporal can provide the Durable Execution layer that makes the whole system reliable without requiring either team to rewrite its own stack.

The framework becomes a per-workload choice, not a one-time commitment.

Human in the loop takes two forms#

“Human in the loop” gets used as though it’s one thing. To a degree, it is: the same pattern applies regardless of who initiates.

Pattern A: the human interrupts the agent#

An operator changes an order mid-delivery, such as a cancellation or a new address. The driver reaches the venue and holds instead of delivering; a human approves the change, and the driver reroutes to Oracle Park. That gate lives in the Workflow, not in an LLM tool. When a human hits stop, you don’t route “stop” through a model.

# The operator’s decision arrives as a Signal.

@workflow.signal
async def resolve_update(self, inp: OrderUpdateInput):
    self._pending_holds[inp.order_id].decision = inp.change_type

# The driver holds at the venue until the decision arrives.

await workflow.wait_condition(
    lambda: self._pending_holds[order.order_id].decision is not None or self._stop
)

Pattern B: the agent asks a human#

ask_human triggers the graph’s own interrupt(). Dispatch runs in its own Workflow, where the child represents the order. The Workflow parks on wait_condition for a Temporal Signal, then resumes the graph with Command(resume=...):

# The reviewer’s decision is signaled to this child Workflow.

@workflow.signal
async def answer_dispatch(self, decision: str):
    self._answer = decision

# Run the graph. ask_human suspends it through interrupt().

result = await compiled.ainvoke(state, config=config)

while result.get("__interrupt__"):
    self._pending_question = result["__interrupt__"][0].value  # Surface for the UI.

    await workflow.wait_condition(lambda: self._answer is not None)

    answer, self._answer = self._answer, None  # Consume and reset.
    self._pending_question = None

    result = await compiled.ainvoke(Command(resume=answer), config=config)

Here’s the pattern worth taking away: the human isn’t special-cased in the control flow. It’s a tool in the agent’s toolset, sitting next to get_fleet_status and submit_dispatch. The agent calls ask_human the same way it calls any tool, on its own judgment. What differs is execution: a normal tool runs as an Activity and returns a value; ask_human suspends the graph and waits on a durable Signal. That’s the whole move. The human is an async API, and you hand it to the agent as a tool.

Who fires that Signal? The dashboard. It is a Temporal client, so when the operator clicks Approve, it looks up the parked Workflow and signals it:

handle = client.get_workflow_handle(child_id)
await handle.signal(LgDispatchWorkflow.answer_dispatch, decision)

And if they never answer? workflow.wait_condition accepts a timeout, and Temporal’s Timers are durable. “Escalate to a backup after four hours” or “auto-reject after a day” is that timeout plus a branch, and the Timer survives a crash the same way the wait does. It is the same primitive, extended.

The human is an async API. Terrible latency, no SLA, and occasionally returns “ask someone else instead.” So you model the wait as durable state the system can hold for days, not a blocked thread or an in-memory promise.

Why “durable” is the load-bearing word#

Human and machine time don’t match. An approval takes four hours or four days. Meanwhile, your cluster deploys daily, pods get evicted, and Workers crash. If the wait lives in process memory, any one of those erases it, and the agent doesn’t fail loudly. It just forgets it was ever waiting, and nobody gets paged.

The industry keeps trying to solve execution reliability, which is deterministic and architectural, with correctness tooling, which is probabilistic and model-shaped. A lost approval isn’t a reasoning failure. It’s a wrong-layer failure. And no eval suite catches it: if the agent forgets it was waiting, the trace just ends. No error, no failed assertion, nothing to grade.

Durable Execution flips it. The Workflow parks on the wait while burning zero compute, and the pending decision lives in the Event History, not RAM, where no deploy, eviction, or crash can touch it. In the demo, I kill the Worker mid-wait, live, and nothing is lost. Restart the Worker, and the approval is still pending exactly where it was; approving it lets the delivery proceed. Because each wait is its own parked Workflow, thousands can wait independently in an open state without consuming Worker CPU. Every decision lands in the Event History, so the audit trail comes free.

The unglamorous parts#

Building it surfaced a few unglamorous truths the framework doesn’t abstract away.

Early on, agent reasoning and driver navigation shared a Task Queue, and the model calls starved the drivers: ice cream melting while the model thought. Separate Task Queues fixed it, one pool for agent reasoning and another for driver Activities, so a slow inference can’t starve navigation.

The driver loops never stop. A Workflow’s Event History grows with every step it takes, so a loop that runs forever needs a way not to accumulate history forever: Continue-as-New. Past an Event History threshold, each driver starts a fresh Workflow Execution, carries its live state forward (position and lifetime delivery count), and keeps going with a clean Event History. It’s the flip side of parking at zero compute, where no Worker holds a looping or waiting Workflow in memory, which is what lets it run for days in the first place.

Multi-agent systems are still distributed systems. The boring rules apply.

The tradeoffs are real. Workflow code runs deterministically, with no wall-clock time or randomness in the Workflow body, but that’s largely what the ADK and LangGraph integrations handle for you: model and tool calls run as Activities, keeping the nondeterministic parts outside the replayable core. You’re also running Temporal alongside your app, self-hosted or on Temporal Cloud, and taking on a learning curve around a few core concepts: Workflows, Activities, and Signals. If your agents never wait on anything slow and never need to survive a crash, you may not need this yet. The moment one waits on a human, it earns its keep.

Two frameworks, one contract#

Each framework does what it’s good at: agent composition and reasoning. Underneath, every model and tool call is recorded as a Temporal Activity that is retryable, replayable, and visible in the log. Kill the Worker mid-reasoning, and Temporal replays from the Event History. A call that already finished returns its recorded result, so no new API call goes out, and you don’t pay for it again.

The edge worth being clear about is that Temporal checkpoints at the Activity boundary, not inside an inference. A call that was in flight when the Worker died can’t resume mid-token, so that Activity retries from the start, and you do pay for that one again. A caveat falls out of those retries: tool calls should be idempotent. A retry that recharges a card or double-books a driver is a bug, so anything with a side effect needs an idempotency key or a deduplication guard.

To be fair to LangGraph, its native interrupt() provides durable execution when paired with a persistent checkpointer, such as PostgreSQL, SQLite, or Redis. That qualifier matters. This demo compiles the graph with an in-memory checkpointer, so the suspended graph lives in process memory and wouldn’t survive a crash on its own. Here, it’s Temporal’s Event History, not LangGraph’s checkpoint, that makes the pause durable. interrupt() suspends the graph; Temporal is what makes it survive.

The distinction isn’t quality; it’s scope. LangGraph’s durability, checkpointer and all, is framework-local: it persists the graph. Temporal is general-purpose Durable Execution: it persists the whole system, including the agent loops, drivers, human wait, and Timers, across both frameworks at once.

Temporal isn’t replacing your framework. It’s the layer underneath it. Run ADK and LangGraph in one system, swap either out, and the durability and human-in-the-loop logic don’t move because they didn’t live in the framework to begin with.

It’s ice cream here, but swap the noun. A refund. A production deploy. A regulated trade. Same gate.

Take it with you#

• Try the demo: github.com/temporal-community/durable-hitl-agents
• Original ADK demo code: github.com/temporal-community/ice-cream-fleet-demo
• 60-second demo video: youtube.com/shorts/Wq7hiN2KYnk