Durable Execution means deciding what survives failure


This is a guest post from Muhammad Abu Bakkar Bin Akmal, Senior Platform Engineer at Xgrid

The question durability has to answer#

When a Kubernetes cluster analysis is interrupted, it can leave work in several states at once. Some analyzers have finished, others are still processing, and one may have failed because a dependency became unavailable. After a restart, the system needs to know which results it can use and which steps still need to run.

An AI agent recommending an upgrade faces a similar problem. A judge has rejected its first proposal, and the Worker stops while the agent is preparing another attempt. Continuing requires the previous proposal, the feedback, and a record of how far the agent got.

Worker restarts, deployments, and temporary outages are routine. Recovering from them gets harder when a process has accumulated progress that the next Worker needs to understand.

Recovering these processes requires more than knowing that a request arrived or reading a status column. The system also needs to answer:

Which analysis steps are already completed, and which ones still need to run?
Was the failed operation safe to retry?
What did the AI agent already propose, and how many model requests remain?
Was the previous recommendation accepted or left unresolved?
What should happen if the Worker disappears again?

That is the difference between an application that merely executes work and one that can remember where the work was when everything around it failed.

This walkthrough examines two workloads in a Kubernetes analysis platform: parallel cluster analysis and iterative AI version recommendation. Drawing on an implementation review, it shows how Temporal preserves progress and where the application still needs explicit policies for retries, cancellation, and accepted results.

Beyond the Kubernetes Job: when "try again" stops being a recovery strategy#

Cluster analysis sounds straightforward. You collect the required context, run a set of analyzers, and combine their results. You persist what you found, and in some cases you trigger an assessment that determines what should happen next.

AI-assisted version recommendation is different. The system may propose an answer, have that answer judged, and incorporate the feedback. It tries again and eventually stops because it reaches consensus or earns approval. It can also stop because it encounters repeated concerns or reaches its configured round limit.

But operationally, both workloads have the same underlying problem: they are processes with memory.

A normal request can often afford to think in terms of:

request → work → response

A durable Workflow cannot. It has to think in terms of:

started → partially completed → interrupted → resumed → evaluated → completed

That distinction becomes important when the infrastructure executing the work is not permanent.

A Worker restart or a deployment that changes configuration can interrupt work already in progress. Recovery gets more complicated when a downstream service becomes unavailable or an Activity times out after the external operation has succeeded. A caller retrying a request may not even know whether the first attempt started. The AI workload adds another wrinkle: a response can have the right structure and still fail the application’s acceptance criteria.

The database alone cannot answer all of those questions. That is why the implementation separates responsibilities:

The database remains the external business read model.
Activities perform external work.
Workflow code owns ordering and coordination.
Event History records the execution state required for replay.
Application policy decides what counts as success, failure, - acceptance, or unresolved work.

The key architectural question becomes:

If this process disappears right now, what must still be known when it comes back?

Figure 1. Two workloads share the same durability questions even though they have no runtime dependency on one another.

Legend: White represents a caller or result. Violet represents Workflow coordination. Teal represents an Activity. Slate represents a data store. Amber represents an external service.

Starting Temporal Workflows safely: the first failure boundary#

The first problem happens before the Workflow even begins. The service already owns a scan record.

A request arrives to create a Kubernetes cluster scan. The service validates the request and creates the pending database record. A caller-supplied option determines whether the service starts the Temporal Workflow.

That sounds harmless. But there are now two different pieces of truth:

The scan record exists.
A Temporal Workflow exists.

Those two events are not automatically the same event.

If the database write succeeds and the Workflow start call fails, or returns an ambiguous response, what should the caller do? Should it create another scan? Update the existing record? Query Temporal first? Assume the Workflow exists?

This is where a seemingly small integration detail becomes a business-state problem.

Give every scan one durable identity#

The implementation uses the business scan ID as the Workflow ID. That is useful because an operator can move between the external record and the Temporal execution using the same identifier.

But identity by itself does not define retry semantics. The start options rely on SDK defaults rather than explicitly defining how to handle an existing Workflow ID. The application should distinguish retries against a running execution from attempts to reuse the ID after an execution has closed. So the architecture gives the team a useful handle, but it still leaves an important policy to define.

Here is the relevant implementation boundary:

Simplified example.

scan, err := persistPendingScan(request)
if err != nil {
    return nil, err
}

startDurableRun := request.Options.StartDurableRun
if startDurableRun {
    options := client.StartWorkflowOptions{ID: scan.ID, TaskQueue: durableTaskQueue}
    _, startErr := temporalClient.ExecuteWorkflow(
        context.Background(), options,
        scanWorkflow,
        scanInput(scan, request),
    )
    if startErr != nil {
        return scan, startErr
    }
}

return scan, nil

The return path matters more here than the ExecuteWorkflow call itself.

If the scan record has been saved but starting the Workflow returns an error, the service returns both the stored record and the error to the caller. The record exists, but the Workflow’s status still needs to be confirmed.

Creating the record and starting the Workflow are separate operations, so one can succeed while the other fails. The platform needs a way to reconcile them: check whether the Workflow started and, if it did not, start it for the existing scan.

If a caller retries after an ambiguous start response, the service needs to know whether it is creating new work or recovering existing work. Without that policy, durable execution alone leaves durable business semantics undefined.

Parallel Kubernetes analysis with Temporal: recovering partial work#

The scan first resolves missing context and handles legacy representations through Activities. Once their completion is recorded in Event History, Temporal can reuse those results during replay rather than repeat the external work.

Six analyzers run together, but failure still has an order#

After normalization, the Workflow runs an add-on detection Activity and then launches six analysis Activities. The six analyzers can execute concurrently. But the Workflow collects their Futures in slice order.

That small implementation detail changes the failure behavior. Imagine analyzer number six exhausts its retries or returns a non-retryable error while analyzer number one is still running. The Workflow waits for analyzer number one before it observes the failure from analyzer six.

The work is concurrent, but the join is not fail-fast.

Simplified example.

policy := &temporal.RetryPolicy{MaximumAttempts: 3}
activityCtx := workflow.WithRetryPolicy(ctx, policy)
detectCtx := workflow.WithStartToCloseTimeout(activityCtx, 30*time.Minute)
analyzeCtx := workflow.WithStartToCloseTimeout(activityCtx, 20*time.Minute)
// The create path currently accepts one resource.
detector := workflow.ExecuteActivity(detectCtx, detectAddons, resource)
if err := detector.Get(ctx, nil); err != nil && !temporal.IsCanceledError(ctx.Err()) {
    return err
}

analyzers := []any{
    analyzeA, analyzeB,
    analyzeC, analyzeD,
    analyzeE, analyzeF,
}

var futures []workflow.Future
for _, analyzer := range analyzers {
    futures = append(futures,
        workflow.ExecuteActivity(analyzeCtx, analyzer, resource))
}
for _, future := range futures { // Slice order; this wait is not fail-fast.
    if err := future.Get(ctx, nil); err != nil && !temporal.IsCanceledError(ctx.Err()) {
        return err
    }
}

There are three separate policies here. Concurrency determines how quickly independent work can proceed. Retries determine how many attempts a failed Activity gets. The join policy determines when the Workflow considers that work unsuccessful. Those are not the same thing.

A retry policy can say "try this Activity up to three times." It cannot say "this scan is still useful if analyzers A, B, C and D succeed but E fails." That is a product decision, and it becomes increasingly important as the number of analyzers grows.

When "complete" doesn't mean complete#

The next problem is more subtle. The Workflow has a business result. The database also has a business status. Those two need to agree.

The implementation uses deferred cleanup to update the scan’s status, but that cleanup has two problems. First, it checks whether the Workflow context was cancelled rather than inspecting the returned error. An ordinary failure can therefore leave the scan marked complete.

Second, it schedules the cleanup Activity on a disconnected context but waits for it on the original context. If that original context is cancelled, the wait can return before the status update finishes.

That creates a dangerous possibility: the Workflow can fail while the business record says complete. For an engineer, that is an implementation bug. For an operator, it is a misleading dashboard. For a customer, it can become an incorrect answer.

The corrected implementation makes the Workflow outcome the source for determining terminal status, and it schedules and awaits cleanup in the same context.

func scanWorkflow(ctx workflow.Context, input ScanInput) (outcome error) {
    defer func() {
        status := "complete"
        if temporal.IsCanceledError(outcome) {
            status = "cancelled"
        } else if outcome != nil {
            status = "failed"
        }

        cleanupCtx, _ := workflow.NewDisconnectedContext(ctx)
        cleanupCtx = workflow.WithActivityOptions(cleanupCtx, workflow.ActivityOptions{
            StartToCloseTimeout: time.Minute,
            RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 3},
        })

        update := StatusUpdate{ScanID: input.ScanID, Status: status}
        statusWrite := workflow.ExecuteActivity(cleanupCtx, writeStatus, update)
        statusErr := statusWrite.Get(cleanupCtx, nil)
        if statusErr != nil {
            workflow.GetLogger(ctx).Error("Status write failed", "error", statusErr)
        }
    }()

    return runScan(ctx, input)
}

If the cleanup Activity exhausts its retries, the database status can remain stale even though the Workflow has finished. The example logs that failure; monitoring should alert on it, and a reconciliation process should compare the stored status with the actual Workflow outcome and repair mismatches. The status-write failure stays separate from the original Workflow outcome so that a secondary cleanup failure does not obscure the primary reason the Workflow failed.

The mechanism used to report failure must not accidentally change the failure being reported.

Cancellation is a business event too#

Cancellation exposes another edge. If an Activity returns an error because the Workflow was cancelled, the implementation must not continue as though it has a valid result. Otherwise, the next line may attempt to use a result that was never produced.

The corrected path therefore checks the Activity error before dereferencing the returned value.

var resolved *ResolvedContext
future := workflow.ExecuteActivity(resolveCtx, resolveContext, resource)
if err := future.Get(ctx, &resolved); err != nil {
    return err
}

if resolved == nil {
    return fmt.Errorf("Context resolution returned no result")
}

resource.Path = resolved.Path
resource.Location = resolved.Location
resource.Compressed = resolved.Compressed

The first scan has a second job#

Once the analysis results are available, the Workflow performs another business decision: is this the first analysis? If it is, the platform can continue to the upgrade-assessment operation.

The order matters. Core results are persisted before the assessment call on the normal write path. So if the downstream assessment fails, the analysis result does not disappear with it. That creates a useful business boundary: analysis persistence and assessment success are separate outcomes. The system can know what it learned even if the next service is unavailable.

Figure 2. Results are persisted before the conditional assessment operation. The downstream failure therefore does not erase the completed analysis.

Legend: Participant bands identify roles. Pale red notes mark cautions.

The corrected implementation also moves the assessment endpoint into Workflow input rather than reading it directly from the process environment state.

var isFirstAnalysis bool
firstCheck := workflow.ExecuteActivity(checkCtx, checkFirstAnalysis, input)
if err := firstCheck.Get(ctx, &isFirstAnalysis); err != nil {
    return err
}

if !input.DryRun {
    write := workflow.ExecuteActivity(writeCtx, persistResults, input)
    if err := write.Get(ctx, nil); err != nil {
        return err
    }
}

if isFirstAnalysis {
    nexus := workflow.NewNexusClient(input.AssessmentEndpoint, "assessment-service")
    operation := nexus.ExecuteOperation(
        ctx, "assess-cluster",
        input.AssessmentRequest, workflow.NexusOperationOptions{},
    )
    if err := operation.Get(ctx, nil); err != nil { // Drop only non-cancellation errors.
        if temporal.IsCanceledError(err) || temporal.IsCanceledError(ctx.Err()) {
            return err
        }
        workflow.GetLogger(ctx).Warn("Assessment failed", "error", err)
    }
}

The implementation intentionally continues after non-cancellation assessment errors. That can be a valid business choice. The important question is whether the product considers assessment to be:

mandatory for a successful scan
advisory after analysis
or an asynchronous follow-on operation

Durable execution makes the sequencing reliable. The business policy is still up to the product.

Coordinating an AI recommendation loop#

The second workload recommends Kubernetes versions through a proposer/judge loop. Each round carries forward the previous proposal, feedback, and remaining budget.

The recommendation Workflow receives:

a project
a project version
Kubernetes versions
control-plane information
source data
prompt configuration
model configuration

The caller is written in Go. The Workflow is implemented in Python. That creates a new durability boundary: the serialized contract between the two languages.

Go checks the caller’s types, and Python can validate the received data. Those checks do not establish that both sides agree on field names, types, and serialization. The shared contract needs its own test.

Both known start paths append a random UUID to the Workflow ID. Repeating the same business request therefore starts a new, distinct execution rather than colliding with an existing one, and the Workflow ID alone cannot answer whether a given project and version combination has already been evaluated. That question is answered by the cache.

A cache hit isn't automatically a good decision#

Caching an AI recommendation sounds like an optimization. It can also become a correctness problem.

The cache key combines:

request fields
source-data version
prompt version
model version

That is the right direction because an answer is only reusable if the inputs that materially shaped it are still equivalent. But there is a second question: was the answer actually accepted?

A response can be perfectly valid according to its schema and still be rejected by the Workflow. For example, the agent may reach the round limit while concerns remain. The result can still have valid fields. It can still deserialize correctly. It can still look like a recommendation. But it was not accepted.

If that unresolved answer enters the reusable cache, a future request may receive it immediately without repeating the evaluation that originally rejected it. That changes the meaning of the cache from "previously accepted recommendation" to "previously produced recommendation." Those are not equivalent.

The corrected cache policy makes acceptance part of the cache contract.

source = await call_activity(load_source_data)
key = canonical_digest({
    "cache_policy": "accepted-v2",
    "request": input.request_fields(),
    "source_version": canonical_digest(source),
    "prompt_version": input.prompt_version,
    "model_version": input.model_version,
})
accepted_reasons = {"consensus", "approved_score"}
cached = await call_activity(cache_lookup, key)
if cached:
    record = CachedDecision.model_validate(cached)
    if record.exit_reason in accepted_reasons:
        return Decision(
            Recommendation.model_validate(record.output), record.exit_reason
        )
decision = await recommend(input, source)
if decision.exit_reason in accepted_reasons:
    record = CachedDecision(
        output=decision.output.model_dump(),
        exit_reason=decision.exit_reason,
    )
    await call_activity(cache_write, CacheWrite(key, record))
return decision

Python-style pseudocode. call_activity stands in for workflow.execute_activity with explicit options, and canonical_digest represents hashing a canonical serialization.

Both the cache-hit and newly evaluated paths return a Decision containing the recommendation and its exit reason. Downstream consumers can distinguish accepted recommendations from unresolved ones. The cache stores and reuses only accepted decisions; syntactic validity alone is not enough.

Older cache records require equivalent treatment through migration or a new namespace so that unresolved recommendations cannot silently bypass the new rule.

The agent has a budget, and the budget isn't the whole cost#

On a cache miss, the recommendation Workflow enters a proposer/judge loop. The proposer has limits on:

model requests
tool calls
rounds

The judge evaluates the proposed result. The Workflow then decides whether to:

accept it
try again
stop because concerns repeat
or stop because the maximum number of rounds has been reached

This is where the distinction between execution failure and business non-acceptance becomes critical. A recommendation that reaches the round limit can still be a legitimate structured outcome, rather than a failed Workflow: "we could not reach an accepted recommendation within the configured process." That is very different from "the Worker crashed," "the model API timed out," or "the output was invalid." Each requires a different operational response.

Figure 3. The recommendation loop has application-level budgets and multiple explicit exits. Accepted and unresolved results must not be treated as equivalent.

Legend: White represents input or output. Violet represents Workflow decisions. Teal represents external execution through Activities. Slate represents a data store. Green means accepted. Red means an error or unresolved result.

Four ways the recommendation can end#

The loop can return four decision outcomes. Budget exhaustion and invalid structured output are handled separately as execution errors.

Consensus. The model and judge reach agreement.

Approved score. The judge approves the recommendation and its score reaches the configured threshold.

Repeated concerns. The system detects the same concerns across rounds and stops rather than continuing indefinitely.

Round limit. The Workflow reaches its maximum configured rounds without acceptance.

Those outcomes should be visible to downstream consumers. Otherwise, an operator sees only "recommendation completed." That sentence hides the most important business distinction: did the system produce an accepted recommendation, or did it stop because it could not reach one?

The corrected implementation makes the exit reason explicit.

requests_used = tools_used = 0
state, previous_concerns = ConversationState(), None
for round_number in range(1, input.max_rounds + 1):
    remaining = Limits(input.request_limit - requests_used,
                        input.tool_limit - tools_used)
    try:
        if remaining.requests <= 0 or remaining.tools <= 0:
            raise BudgetExceeded()
        result = await run_proposer(input, state, remaining)
    except BudgetExceeded:
        raise ApplicationError("Proposer budget exhausted",
                                type="UsageBudget", non_retryable=True)
    except InvalidStructuredOutput:
        raise ApplicationError("Invalid structured output",
                                type="InvalidOutput", non_retryable=True)
    requests_used += result.usage.requests
    tools_used += result.usage.tool_calls
    judged = await workflow.execute_activity(
        judge_round,
        args=[result.output, state.judge_history],
        start_to_close_timeout=activity_timeout,
    )
    state.update(result, judged)
    verdict = judged.verdict
    if verdict.consensus:
        return Decision(result.output, "consensus")
    if verdict.approved and verdict.score >= input.score_threshold:
        return Decision(result.output, "approved_score")
    if round_number > 1 and verdict.concerns == previous_concerns:
        return Decision(result.output, "repeated_concerns")
    previous_concerns = verdict.concerns
return Decision(result.output, "round_limit")

Python-style pseudocode. run_proposer represents the durable agent integration and its remaining allowance; its exception names stand for budget and output-validation failures.

The loop makes another important distinction. The proposer budget does not automatically represent the total inference cost. Judge calls sit outside those counters. So if the business wants a true "maximum inference spend" guarantee, the judge needs its own accounting policy. Again, Temporal provides the durable mechanism. The application still defines the economics.

The configuration problem nobody sees until replay#

There is another subtle durability concern. Some of the recommendation configuration comes from Worker-process state:

model identifier
round limit
score threshold

That is dangerous when those values can affect Workflow decisions. Imagine:

Workflow starts with model version A.
Worker is redeployed.
Worker configuration now points to model version B.
The Workflow replays.

If the Workflow derives its behavior from mutable process configuration rather than recorded input, the same Workflow can encounter different configuration after replay. The corrected pattern therefore pushes important decisions into Workflow input.

The goal is concrete: the same Event History needs to mean the same thing tomorrow that it meant today.

The Worker is part of the product#

The Workflow code is only one part of the system. The Worker registers:

one Workflow
four Activities
an agent plugin
a Nexus service handler

The recommendation Workflow's caller uses the direct start path. The Worker also registers a Nexus service handler as an additional interface.

Infrastructure provisions the Worker environment, including:

mTLS configuration
logging
data access and cache access
an inference permission

A durable Workflow reduces the amount of recovery code the application needs, though operational responsibility still stays with the team running it.

The hidden asset: Event History becomes operational data#

Event History lets operators inspect how a Workflow unfolded without piecing together its progress from scattered logs. For a long-running analysis or recommendation, that makes it easier to investigate where execution stopped and which results were recorded.

Four failure modes worth turning into tests#

The implementation review ultimately produces a more useful artifact than a list of code observations. It produces testable business scenarios.

Failure scenario	What the system should prove
Worker configuration changes	Replay preserves the intended assessment and recommendation configuration
Workflow fails during execution	Stored business status matches the actual Workflow outcome
AI recommendation is unresolved	An unresolved recommendation is neither cached nor returned from the cache as an accepted result.
Context resolution is cancelled	The Workflow returns the cancellation before attempting to use an unset result

These function as contracts between the platform and its operators. A strong test should answer: if this happens in production, what will the business see?

For the recommendation Workflow, each reachable exit should also have its own test. For the scan Workflow, several areas need dedicated coverage:

cancellation
retry exhaustion
Activity fan-out
query behavior
terminal status

Turn those gaps into tests that assert the stored status, returned result, exit reason, and cleanup behavior.

What to build before calling this production-ready#

The implementation suggests a practical sequence.

1. Start with one controlled Workflow#

Pick one cluster-analysis path or one recommendation path. Pin the Temporal SDK and agent integration versions. Use deterministic test data and stubbed external services. The goal is to make the first recovery scenarios reproducible.

2. Define the start contract#

Write down what should happen in each of these cases:

the database record exists
Workflow start fails
Workflow start returns ambiguously
the caller retries

Define the Workflow ID conflict and reuse policy. For the Go-to-Python path, test the serialized request contract directly.

3. Make every external effect retry-safe#

Put external I/O behind Activities or the supported durable integration. Then ask: if this Activity succeeds and the Worker dies before completion is recorded, what happens when it runs again? If the answer is "the external system gets a duplicate," add a stable business key and deduplication strategy.

4. Define failure and acceptance separately#

For every operation, decide:

Is it retryable?
How many attempts?
How long can one attempt run?
Is it required for the overall result?
What happens if it is cancelled?
Does its output count as accepted?

For AI Workflows, keep three states explicit: execution error, unresolved result and accepted result. They are different business states.

5. Test the outcome, beyond the exception#

A test that says "the Workflow returned an error" may not be enough. Also test:

the database status
the cache
the returned business result
the exit reason
cleanup behavior
and what an operator would see

6. Make Workflow state operationally visible#

Correlate business IDs with Workflow IDs and Run IDs. Track Activity failures and Task Queue delays. Expose terminal states and recommendation exit reasons. Alert on records that exist without confirmed executions. A durable system should make recovery easier for the code, and just as much for the people operating it.

7. Simulate Worker loss#

Stop a Worker during an Activity. Restart it. Simulate a downstream outage. Allow an Activity to complete externally and then fail before completion is recorded. Then verify the business result. This is where Durable Execution either proves its value or exposes the remaining assumptions.

8. Establish ownership before launch#

Verify:

Task Queue routing
Worker registration
credentials
data access
history growth
long-running conversations
alert ownership
incident procedures
and deployment/replay readiness

The snapshots do not establish how callers populate the start option or whether a caller uses the registered Nexus handler. The reviewed tests cover empty-resource validation and context-resolution failure; no coverage was found for the failure scenarios above, a dedicated recommendation Workflow test, or a live end-to-end Nexus test. Examples replace internal types and identifiers. Examples labeled "Corrected for publication" illustrate the changes described rather than reproduce the reviewed implementation.

The real value of Temporal is what it remembers#

Without durable orchestration, recovery tends to accumulate. A status column gets added. Then a retry flag. Then a reconciliation job. Then a scheduled sweeper. Then a deduplication check. Then a second status field to distinguish "processing" from "really processing." Then another script to identify records that have been stuck for more than an hour.

Each addition solves a real incident. Over time, the business process becomes scattered across databases, queues, logs, cron jobs, and retry wrappers.

The two workloads examined here point toward a clearer division of responsibility. Workflow code owns coordination, Activities perform external effects, and Event History records what the Workflow needs to recover. Business records expose the result, while application policy defines what counts as accepted, failed, cancelled, or unresolved.

After a Worker restart, Temporal reconstructs Workflow state from Event History and continues execution. Activities whose completion was not recorded may run again, so external effects still need retry-safe behavior. A Kubernetes analysis can retain its progress, and an AI recommendation can carry its conversation and remaining budget across rounds. Callers receive the exit reason, and the cache reuses only accepted recommendations.

Operators can see why a scan failed or why a recommendation was rejected, then act on that result. Temporal preserves the progress needed for recovery; the application’s policies determine how to proceed.

Your service may already run Temporal. Its boundaries around retries and cancellation may still be unclear, and the same may be true of configuration and recovery. Xgrid can review a critical Workflow against those failure modes through a Temporal 90-Day Production Health Check.

If you are preparing a new Workflow for production, a Temporal Launch Readiness Review can examine the execution model and failure policies. It can also cover test coverage and operational ownership before launch.

Frequently asked questions#

Does Temporal prevent duplicate side effects?#

No. Activities can run again after failures. An external write can succeed before the Activity completion is recorded, allowing the Activity to execute again. For effects that must not happen twice, use stable business keys and deduplication at the receiving system. A Workflow ID alone does not make an Activity idempotent.

Can Temporal orchestrate AI agents written in another language?#

Yes, provided the Workflow and caller agree on the serialized contract. A Go caller can start a Python Workflow using a registered Workflow name and compatible serialized input. Test that shared contract independently to confirm that both sides agree on field names, types, and serialization.

Should every valid AI response be cached?#

Not necessarily. Schema validity only proves that the output has the expected structure. It does not prove that the Workflow accepted the recommendation. For decision-oriented AI systems, cache policy should distinguish accepted output from unresolved output and enforce that rule on both cache reads and writes.

What should be tested before deploying these Workflows?#

At minimum, test:

ambiguous Workflow starts
duplicate side effects
Activity retries
retry exhaustion
cancellation
terminal status
every reachable recommendation exit
cache acceptance rules
replay after configuration changes
cross-language contracts
Nexus boundaries
and Worker-loss recovery

The most valuable tests are the ones that assert the business outcome, well beyond the raised exception.