How Sherlocks AI uses Temporal to orchestrate AI agents for incident resolution

Akshat Sandhaliya is CTO and co-founder of Sherlocks AI. He was previously Head of AI at Doubtnut, the largest education app in South Asia, and is building AI agents to handle the 3am pages so engineers don’t have to.

Sherlocks AI is an AI-powered SRE platform. When a production incident fires, we dispatch a fleet of specialized AI agents that investigate autonomously, the way a seasoned on-call engineer would: pulling up dashboards, querying observability tools, and tracing the problem back to root cause. The goal is to bring Mean Time to Resolution (MTTR) down from hours to minutes. We do that today for customers including Fynd, Topmate, Wafeq, and TradeIndia.

The hard part of building this turned out not to be the AI. It was making the agents run reliably. A single incident can trigger 10 to 15 agents working in parallel across the full resolution lifecycle: identification, investigation, root cause analysis, and remediation. Each agent makes multiple LLM calls and queries customer infrastructure across different observability tools, each with its own API and query language. A complex investigation can involve hundreds of steps over 15 to 20 minutes, with dozens of points where something can fail. When you’re debugging someone else’s production incident, your own platform failing silently is the worst possible outcome. An SRE waiting five minutes only to see the agent crash without a result isn’t just a bad experience. It erodes the trust that makes autonomous incident response possible at all.

This is how we ended up running all of it on Temporal, what we tried before, and what we picked up along the way.

The problem: Four Workflows, one system#

Our platform runs four distinct kinds of work, and each one has to be reliable.

Knowledge Graph construction. Every few minutes, per tenant, we ingest tribal knowledge from team conversations, pull metrics from monitoring systems, map infrastructure topology, and index runbooks and documentation. That gives our agents context before an incident even fires and eliminates the discovery tax.
Infrastructure scanning. A full scan of a customer’s cloud environment can take 30 minutes to two hours for enterprise accounts because of API rate limits and quotas, so retries and checkpointing are essential rather than optional.
AI agent investigations. This is the core product. A growing roster of specialized agents covering databases, Kubernetes, logs, cloud infrastructure, APM, and more gets dispatched across every phase of incident resolution: identification, investigation, root cause analysis, and remediation. Many run in parallel, sometimes with a human stepping in on complex cases.
Event ingestion. Slack events, GitHub webhooks, and other triggers have to be durably acknowledged before they enter a processing queue. A dropped webhook might represent a real production incident, so we can’t afford to lose one.

All four are long-running and failure-prone. The question was whether to build separate infrastructure for each or find one system that could handle all of them.

What we tried before#

We worked through the obvious options in order, and each one fell short at the same place.

Threads came first. They were great in demos, but in production, the server restarted and every in-flight investigation vanished, with no checkpoint and no record the work ever happened. Message queues came next. They were fine for simple event-driven work, but when a 45-minute infra scan failed at step 37, the only options were to restart from scratch or build our own checkpointing layer on top. Agent frameworks were the third stop. They solve the “how do I structure agent logic” problem well, but not ours: no Durable Execution, no real Retry Policies distinguishing retryable from non-retryable errors, no visibility into what’s running, and no support for work that takes hours instead of seconds.

That left building it ourselves: a message queue for events, cron for schedules, a state machine for scans, a custom orchestrator for agents, a Redis checkpointing layer, and a monitoring stack to catch when any of it got stuck. Five or six systems to build and keep alive, none of which made our agents any smarter. At an early stage, every week on plumbing is a week not on the product. So we stopped and made a deliberate design choice instead.

There was another constraint we didn’t appreciate at first: most agent frameworks lock you into a specific model provider. We don’t use a single model. We run an ensemble, with different models for different agents, chosen based on benchmarks we run against each agent’s task. A log analysis agent might perform best on one model, while a Kubernetes investigator works better on another. Cost matters too. Some agents run on every alert, others only on escalations, and the token economics are different for each. We even offer customers model selection by tier because in mission-critical SRE work, both time-to-output and cost-per-token are real constraints. Frameworks that tightly couple your agent code to a single provider make all of that harder to do.

With Temporal, none of this is an orchestration problem. The model call is just an Activity. We route all LLM calls through a gateway, so any agent can use any model from any provider. We swap models in config, not in code. When a provider degrades, a new model launches, or a customer’s cost profile changes, we update a YAML file and the next Workflow picks it up. That kind of flexibility matters when the model landscape changes every few months and your agents need to keep running regardless.

The realization that mattered was that agent frameworks and Temporal aren’t competing. They operate at different layers. You can run any agent framework on top of Temporal: the framework handles code patterns and agent logic, and Temporal handles the execution environment, including durability, retries, scheduling, and observability, with no lock-in to either. Instead of maintaining five separate systems, Temporal covered all four of our workflow patterns in one place. That’s not just less operational overhead. It’s the difference between a team that ships agent improvements every week and a team that spends half its time keeping the plumbing alive.

Why we picked Temporal#

A few things made the decision:

Workflows as code. Our investigation logic is complex and changes constantly. Writing it in Python instead of YAML or JSON state machines made a real difference in how fast we could move.
Durable Execution. Every step is persisted, so a Workflow that crashes mid-investigation resumes exactly where it stopped. For infra scans that run over an hour, that property is the reason we chose Temporal at all. But it’s equally critical for AI investigations: an SRE who waits 10 minutes for an agent to analyze their incident cannot afford a silent failure. The agent has to either finish or pick up exactly where it left off. With Temporal, it does, automatically.
Retry Policies. LLM calls fail, customer APIs time out, and rate limits kick in. Temporal’s retry framework is mature: it distinguishes retryable from non-retryable errors, handles exponential backoff, and lets us configure all of it. We didn’t write a line of retry logic ourselves.
Scheduling. Our Knowledge Graph updates every few minutes per tenant, crawling Slack messages, ingesting documentation, and pulling fresh metrics. We also run proactive investigations on a schedule, looking for anomalies before they become incidents. Instead of maintaining cron jobs for long-running scripts, we use Temporal’s Scheduler. It’s one fewer system to operate, and the scheduled Workflows get the same durability, retries, and observability as everything else.
Human-in-the-loop with Signals. Some investigations need a person to step in. Signals let us push input into a running Workflow without building custom polling or webhook infrastructure.
Observability. When a customer says an investigation seems stuck, we open the Temporal UI and see which Activity is running, what it’s doing, and whether it’s retrying. For a system where agent behavior is inherently non-deterministic, that visibility matters a lot.
Model agnosticism. Temporal doesn’t care which LLM you call. An Activity can call OpenAI, Gemini, Claude, or a self-hosted model. The orchestration is the same. That let us run an ensemble of models across our agents, benchmark each agent against different providers, and offer customers model selection by tier, all without touching a single line of Workflow code.

How it all fits together#

Our system has four workflow pipelines, all on Temporal Cloud:

A few structural choices hold the rest together:

Per-tenant Workflows. Every tenant gets its own Workflow instances. Combined with Temporal’s Namespace and Task Queue primitives, that gives us strong isolation between customers, with no chance of data leaking across tenants.
Workers as separate processes. We scale by adjusting the number of Worker instances. Each process runs one Temporal Worker with configurable concurrency for parallel Activities. If a Worker goes down, Activities pause at the queue point and resume when another Worker is available. Early on, investigations ran slowly because of Activity contention, and tuning Worker parallelism fixed it.
Codec Server for encryption. All data sent to Temporal Cloud passes through our self-hosted Codec Server and is encrypted before it leaves our environment. We handle customer infrastructure data, so encryption is a requirement, not an option. Temporal’s data encryption docs describe how this works.

The four patterns in practice#

Scheduled Knowledge Graph construction#

The per-tenant Workflows that fetch metrics, crawl Slack, ingest documentation, and update the Knowledge Graph replaced a cron-based setup. Because the Workflows are long-running, having scheduling, retries, and checkpointing in one place made the most sense.

Temporal’s Scheduler turned out to be more useful than we expected. Beyond Knowledge Graph updates, we use it to run proactive investigations: periodic checks that look for slow-moving anomalies like gradually degrading response times or climbing error rates. These are the kinds of problems that don’t trigger alert thresholds but compound into incidents. Having them run as scheduled Workflows means they get the same durability and observability as reactive investigations, with zero additional infrastructure.

Long-running infra scanning#

This is our most demanding Workflow by duration. Each step is an Activity with its own Retry Policy, the Workflow checkpoints as it goes, and we scale scan scope by company size. If a Worker dies at minute 45, the scan continues from that exact step.

AI agent investigations#

This is the most involved pipeline and the one where Temporal’s value compounds the most. When an alert fires, it doesn’t just trigger a single agent. It kicks off a multi-stage process: identification, investigation, root cause analysis, and, in some cases, automated remediation, with different specialized agents handling each phase. A complex incident can involve 10 to 15 sub-agents running across these stages, each making multiple LLM calls and infrastructure queries, all of which need to be durable, observable, and recoverable.

Child Workflows for fault isolation. Each sub-agent runs as its own Child Workflow. If a Database Sherlock hits an unresponsive customer endpoint and crashes, only that child fails. The parent keeps running, collects findings from the agents that succeeded, and synthesizes whatever it has. We set ParentClosePolicy.TERMINATE on every child, so when the parent investigation is cancelled, say because the alert auto-resolves, all child agents are torn down automatically. No zombie agents burn LLM credits in the background.

The parent orchestrator manages its children through a bounded concurrency pool: we cap parallel sub-agents at a configurable limit and queue the rest. That keeps LLM cost and API pressure predictable per investigation without changing agent code.

Signals as a coordination backbone. We started using Signals for human-in-the-loop and ended up using them for everything. Child agents signal their findings back to the parent as they go, not just at completion. The parent forwards confirmed findings to a synthesis agent in real time. Agent logs and step-by-step progress flow back through Signals too, giving our frontend a live view of what each agent is doing.

This creates an event-driven orchestration loop: the parent Workflow sits in a wait_condition, wakes when any child signals a result, processes it, and goes back to waiting. No polling, no timers, just reactive signal handling.

Cross-investigation awareness. Production infrastructure is dynamic. Traffic patterns shift, incidents cascade, and multiple alerts often fire within minutes of each other. When several investigations run concurrently for the same tenant, Signals give us a natural mechanism for cross-pollination: one investigation’s confirmed finding can be forwarded to a sibling investigation as additional context, helping agents avoid redundant work and converge on root cause faster. Because every Workflow has a stable ID and Signals are durable, this coordination is reliable even when investigations start and finish at different times.

Human-in-the-loop for stuck investigations. This turned out to be one of our most valuable features.

AI agents are non-deterministic. Sometimes they go down the wrong path, and sometimes they lack context that only a human has. We needed a way for on-call engineers to steer a running investigation without restarting it.

Every investigation Workflow sets a Search Attribute, waiting_for_signal, whenever it’s idle: waiting for sub-agents to report back or pausing between investigation rounds. Our dashboard queries Temporal for Workflows where waiting_for_signal = true, surfacing investigations that are ready for input.

From there, an engineer can:

Inject a new hypothesis. “I think this is a connection pool issue; check RDS.” The orchestrator spins up a new sub-agent to validate it, prioritizing human-submitted hypotheses over machine-generated ones.
Add evidence to a running sub-agent. “The deploy log from 2 a.m. is relevant.” The sub-agent receives the evidence mid-investigation, re-evaluates its findings, and signals back an updated verdict.
Cancel a sub-agent. “That Kubernetes check is irrelevant, stop wasting time.” The child Workflow shuts down gracefully.
End the investigation. “We’ve found it, wrap up.” The orchestrator signals the synthesis agent to produce a final root cause analysis with whatever findings exist.

None of this requires stopping or restarting the Workflow. The investigation keeps its full history and all prior findings. For a system where agents can take anywhere from two to fifteen minutes, being able to course-correct mid-flight instead of starting over saves real time during incidents.

We also expose a Query handler that returns the live state of every hypothesis in an investigation: which are being validated, which are confirmed, and which failed. The dashboard polls this to show real-time investigation progress. Unlike Signals, Queries are read-only and don’t modify the Workflow, making them safe to call at any frequency.

Smart retries for LLM calls. LLM calls fail in ways that are different from typical API calls. A timeout doesn’t necessarily mean the request was bad. It might mean the model is overloaded, or the reasoning chain is legitimately long. Our model inference Activity uses Temporal’s retry mechanism with a twist: on the first retry, it drops expensive inference parameters like reasoning effort. On subsequent retries, it falls back to a secondary model entirely, a cheaper, faster alternative that still produces a valid result. Because we benchmark our agents against multiple models, we know which fallbacks are acceptable for each task. All of this is configured in the Activity’s Retry Policy. The Workflow code doesn’t know or care which model ultimately answered, and the investigation keeps moving instead of stalling on a single provider’s bad day.

Durable event ingestion#

Once Temporal accepts an event into a Workflow, it runs to completion, so Slack events and GitHub webhooks get acknowledged and processed reliably without anything falling through.

Versioning: Something we learned the hard way#

Temporal’s determinism model means that when we deploy new agent code, Workflows already running keep following the code paths they started on. A 45-minute investigation won’t jump to new, untested logic halfway through. That behavior is intentional and useful, but it caught us off guard.

We iterate on agent patterns constantly. Without explicit version branching in the Workflow code, an old investigation just keeps running the version it began with. Temporal has versioning primitives that handle this, but you have to plan for them from the start. It doesn’t happen on its own.

Why Temporal Cloud#

We run on Temporal Cloud rather than self-hosting. At our stage, time spent on Temporal’s persistence layer, History Service, and Matching Service is time not spent making our agents better.

Scaling is cleaner too. We adjust Worker count, and the orchestration, queuing, scheduling, and persistence are handled for us. Going the Kafka and self-hosted route would have made scaling a much larger project.

There’s also security. Temporal Cloud plus our Codec Server means all Workflow data is encrypted, which matters when you’re handling customer infrastructure telemetry.

What we’ve learned#

Durability is table stakes for SRE AI agents in production. A demo agent can crash and you restart it. A production agent an SRE is depending on at 3 a.m. cannot. Silent failures destroy the trust that separates a tool people use from one that sits on a shelf. Temporal’s Durable Execution meant we never had to build crash recovery, checkpointing, or “is it still alive?” monitoring.

One system instead of four. We could have run cron for scheduling, Kafka for events, an agent framework for the AI, and something else for long-running scans. Temporal covers all four, and the drop in operational overhead alone made it worth it.

No model lock-in. Temporal doesn’t care which LLM you call inside an Activity. We run an ensemble, with each agent benchmarked against providers for the best balance of accuracy, cost, and speed. We fall back to secondary models on failure and swap providers when the landscape shifts, all without touching Workflow code. When you’re running dozens of agents in production, that decoupling is how you keep the system economically viable.

Heartbeating matters for AI agents. LLM-based agents vary a lot in how long they take. A Database Sherlock might need two minutes or twelve depending on the data. Heartbeating is how you tell “still working” from “stuck” without setting absurd timeouts.

Build for partial results. Not every agent succeeds in every investigation. Some hit permission issues, and some turn out to be irrelevant. Our synthesis step works with whatever findings come back instead of failing because one agent timed out.

Plan for versioning early. Temporal’s deterministic replay means code changes don’t apply to running Workflows. If your agent logic changes as often as ours does, build your deployment strategy around that from the start.

Mind idempotency. Temporal retries re-execute Activities from scratch. For LLM calls, that’s fine: you get a different but equally valid response. For side effects like creating Slack threads or posting RCA updates, you need idempotency keys.

Where we’re headed#

Agents that watch your system continuously, not only when an alert fires. Picture a Sherlock that notices your database connection pool shrinking 2% a day, correlates it with a config change from last week’s deploy, and opens a PR to fix it before any customer feels it. Temporal’s Scheduler already powers our proactive anomaly checks. The step from “check every few minutes” to “continuously watch and act” is an engineering challenge, not an infrastructure one. The infrastructure is already there.