Stop restarting your tests

Large test suites have the annoying habit of failing unexpectedly.

We all start with the best of intentions to keep the tests small and focused, but they grow. And grow. And grow. And before we know it, we have thousands of tests. We run the tests locally and they all pass. Then we commit our changes, only for them to fail in our CI pipeline. Maybe a network issue caused a page to load a fraction slower. Maybe an external service disappeared. Maybe the machine ran out of memory.

You rerun the pipeline. And this time, everything passes.

For a unit test suite, that’s mildly irritating. For an end-to-end suite with headless browsers that takes hours to complete, it can mean losing an entire feedback cycle. If it’s your nightly test suite, you might not discover the failure until the next day and lose significant development time.

The problem isn’t always that the tests are badly written. Some tests are inherently vulnerable to such unreliability: browsers, networks, external services, infrastructure, timing and the like. Even the most well-designed test suites eventually encounter transient failures.

What if a failed test didn’t mean starting over? That’s the question I wanted to explore by using Temporal to orchestrate a Playwright end-to-end test suite.

The result is a reference implementation that uses Temporal to turn a test suite into a collection of independent, durable units of work.

Playwright is the demo. The interesting part is the orchestration.

Make progress durable#

Most CI systems treat a test suite as a single job.

The job starts, the tests execute and eventually the job succeeds or fails. If it fails, we rerun it. Even if most of the tests have already passed.

There are various ways of making this less painful. Test runners can retry individual tests. We can split suites into shards and then the CI system can run those shards in parallel. We can keep throwing bigger runners at the problem.

These all help, but I wanted to look at the problem from a different angle. Instead of treating the test suite as one big unit of work, what if we treated each independent test execution as its own unit of work and made progress durable?

That’s a problem Temporal already knows how to solve.

In the example, a client discovers the tests that need to run and builds an execution plan. For each Playwright spec, it creates an execution for Chromium, Firefox and WebKit.

Each of those executions becomes an independent Temporal Activity, orchestrated by a Workflow.

Conceptually, it looks something like this:

The Activities can run concurrently, but the more important thing is that they succeed or fail independently. Imagine the Chromium and WebKit executions complete successfully, but Firefox fails because of an intermittent problem.

We don’t need to run Chromium or WebKit again.We only need to retry Firefox.

The other tests have already completed successfully and we’ve made progress. Why would we throw that away?

Temporal does not make flaky tests less flaky#

Temporal does NOT fix your tests.

If a test is broken, it must fail. If the application has a race condition, Temporal won’t make it disappear. If your tests aren’t independent and rely upon state created by previous tests, changing the execution order will still bite you.

What Temporal does is give us a way to deal with failures in the execution of the test suite.

Each Activity in the example has its own Retry Policy. If one fails because of an intermittent problem, Temporal can retry that execution without affecting any of the work that has already completed.

If the retry succeeds, we carry on.If the retries get exhausted, the Workflow fails and we can see which execution failed and why.

So, instead of:

"Something failed. Run everything again."

We get:

"The Firefox execution for /path/to/test.spec.ts failed. Retry that bit."

Retries themselves aren’t especially interesting. Test frameworks and CI systems have supported retries for years. The key thing is that the retry is part of a durable execution. Temporal knows what has completed, what is still running and what needs to happen next.

We don’t have to reconstruct the state ourselves. Indeed, we don’t even have to change our test suite to make this happen.

Temporalising the test suite#

There is one constraint I had when looking at this problem: there must be no change to how the tests themselves work.

When developing a feature, I still want to be able to run:

playwright test

To my mind, there’s no benefit from this if we butcher the test runner to use Temporal. That merely replaces one set of known problems (tests failing intermittently) with a bigger set of unknown ones. We’d have to document how our custom version of Playwright behaves and maintain it independently of the open-source community. Playwright is a mature, well-documented and fully functional test framework. I don’t want to throw that away.

Temporal sits around the test suite, rather than inside it.

The existing Playwright configuration and tests remain normal Playwright. The Temporal-specific code lives separately under tests/temporal.

This wrapper has three jobs:

1. Discover the work we want to execute
2. Use a Workflow to orchestrate that work
3. Use Activities to invoke the test runner

The same idea applies regardless of the test framework or language it’s written in.

The command might be go test, mvn test or something entirely different. Temporal doesn’t need to understand the test framework. It needs a way to describe a unit of work and an Activity capable of executing it.

Discovering the work#

The first job is to decide what the individual units of work are.

For this demo, I’ve chosen one Playwright spec running against one browser. A work item is therefore just data describing those two things:

interface BrowserExecution {
  spec: string;
  project: string;
}

For two spec files and three browser projects, we end up with six executions:

• auth.spec.ts + Chromium
• auth.spec.ts + Firefox
• auth.spec.ts + WebKit
• unauth.spec.ts + Chromium
• unauth.spec.ts + Firefox
• unauth.spec.ts + WebKit

The client discovers these before starting the Workflow and passes the complete execution plan as Workflow input.

This is by design. Discovering tests means interacting with the filesystem and potentially executing Playwright commands. These are non-deterministic operations and don’t belong inside Workflow code. In principle, you could have a discoverTests Activity at the start of the Workflow, but for this CI setup, I think it's cleaner to perform discovery before starting the Workflow. This will run in a CI pipeline where we already have the repository and know which files are on disk, so there’s little benefit in making discovery part of the durable execution.

This setup also means the Workflow doesn’t need to understand Playwright. It receives a list of things that need doing and has one job: make sure that they get done.

There’s nothing particularly important about my choice of spec and browser as the unit of work either. A larger suite might use Playwright shards, tagged groups of tests or some other independently executable unit.

The important question is: what is the smallest useful piece of the test suite that can safely run on its own?

Orchestrating the work#

Once we’ve identified the work, we start a Workflow with the execution plan as its input.

The Workflow fans out across those executions and schedules an Activity for each one. In simplified form, the important bit looks like this:

const settled = await Promise.allSettled(
  executions.map((execution) =>
    executeBrowserProject(execution),
  ),
);

Using Promise.allSettled is intentional. If six executions are running and one eventually fails, I don’t want that failure to stop the other five. They might all be perfectly healthy, so we don’t want to throw that away.

The Workflow waits for every execution to reach a terminal state and can then determine whether the overall test run was successful.

You’ll also notice how little the Workflow is actually doing. It doesn’t start Playwright, read files, launch browsers, allocate ports or even know how a test gets executed.

That’s Activity work. The Workflow is just the durable coordinator.

Activities invoke the existing test runner#

This is the boundary that makes the pattern useful outside this particular demo.

An Activity receives a work item, such as:

{
  "spec": "tests/e2e/auth.spec.ts",
  "project": "firefox"
}

and turns it into an invocation of the existing test runner.

Conceptually, that’s nothing more than:

playwright test tests/e2e/auth.spec.ts --project=firefox --workers=1

That’s it.

We’re not implementing a browser test runner inside an Activity. We’re invoking the one we already have.

If this were a Go project, an Activity might invoke:

go test ./some/package

The same applies to Java, Ruby, Python or anything else that can be invoked from an Activity.

The Activity is an adapter between a Temporal unit of work and whatever command already executes that work.

This is why the test suite itself can be completely unaware of Temporal. The tests don’t import the Temporal SDK. They don’t know whether they’re being executed by a developer on their laptop or by an Activity on a remote Worker.

We’re adding orchestration around the tests, not rewriting them.

Making executions independent#

There’s a practical problem once we start running these Activities concurrently: they must not trip each other up.

The application in this demo runs on a known port. That’s fine when running playwright test locally, but not if several Activities try to start their own copy of the application on the same Worker at the same time.

Each Activity finds an available TCP port and passes that into the application and Playwright configuration.

Conceptually:

When Temporal isn’t involved, the normal defaults still apply, so I can still run playwright test locally without caring about any of this.

Ports happen to be the shared resource in this demo, but every test environment will have its own version of the problem. It might be database schemas, temporary directories, containers, test accounts or some other piece of shared state.

If we’re going to execute units of work concurrently, and potentially on different Workers, they need to be sufficiently independent to do that safely.

This is also why Temporal cannot magically fix a test suite whose tests depend on one another. If test B requires test A to have run first, arbitrarily distributing them across Workers isn’t going to end well. That’s a property of the test suite we need to understand before deciding how finely to split the work.

In practice, tests that build on state created by previous tests are already problematic. If test #6,481 fails because of something that happened hundreds of tests earlier, isolating the cause is difficult, especially when running #6,481 on its own works perfectly.

Temporal doesn’t create that problem, but distributing the tests will expose it.

Choosing the retry boundary#

Now we have something useful to retry.

One Activity represents one independently executable piece of the existing test suite. In this example, that’s a Playwright spec running against a particular browser. Each Activity has a Retry Policy. If Playwright exits successfully, the Activity completes. If the Activity reports a failure, Temporal can retry it according to that policy.

Crucially, the retry happens at the boundary we chose earlier.

If tests/e2e/auth.spec.ts + Firefox fails, that's the only execution we retry.

We don’t restart Chromium or WebKit. And we certainly don’t restart the entire test suite.

There can actually be two retry mechanisms at play here. Playwright has its own support for retrying individual tests, while Temporal can retry the larger Activity if the Playwright command ultimately fails.

These mechanisms operate at different levels.

Playwright understands the individual test and has much more context about what happened inside it. Temporal understands the larger piece of orchestrated work and whether that work ultimately completed.

Depending on the test suite, you might use both. You might let the test runner handle some failures and Temporal handle others. You might also decide that certain failures should not be retried at all.

The important thing is that we’ve chosen the retry boundary deliberately rather than falling back to “rerun the CI job”.

Scaling the execution#

Splitting the suite into independent Activities also gives us another useful property: we can distribute the work.

Activities are delivered through a Temporal Task Queue. Any Worker polling that Task Queue can execute them.

With one Worker, we have one pool of execution capacity. Add more Workers and we add more capacity.

The Workflow doesn’t change. The tests don’t change.

The Workers simply compete for work from the same Task Queue.

For this demo, I use standard GitHub-hosted runners. Each runner starts a Temporal Worker polling the same Task Queue, allowing the browser executions to be spread across several machines.

This is mainly for convenience. I wanted somebody to be able to fork the repository and see the pattern working without first needing a Kubernetes cluster or a fleet of dedicated test machines.

There are some compromises as a result. In particular, the Workers in the demo monitor the Workflow so they know when to shut themselves down. That’s useful when a Worker is itself running inside a GitHub Actions job, but it’s not how I’d normally manage the lifecycle of a long-running Temporal Worker.

In a production test platform, the Workers could be running on Kubernetes, virtual machines, containers or whatever infrastructure is appropriate for the workload.

The Workflow doesn’t care where they are. Neither does the test suite.

What happens when a Worker disappears?#

Once we distribute work across machines, those machines become another thing that can fail.

A runner might disappear. A Worker process might crash. A machine might be replaced halfway through an execution.But the state of the overall test run doesn’t live in those Workers.

Temporal knows which Activities have completed and which are still outstanding. Subject to the timeouts and Retry Policy we’ve configured, work that doesn’t complete can be retried and picked up by an available Worker.

Again, the goal isn’t to pretend failure doesn’t happen. The goal is to stop one failure from throwing away unrelated progress.

Try it yourself#

The complete example is available in the temporal-sa/temporal-playwright-example repository.

The application being tested is deliberately boring. It exists to give Playwright something to interact with while keeping the focus on the orchestration.

The demo also includes a way to deliberately introduce intermittent failures. This isn’t intended to accurately simulate real-world test flakiness. It simply means you don’t have to sit around waiting for something to break before seeing Temporal’s retry behaviour.

Run the Workflow and the individual spec/browser executions fan out across the available Workers. Introduce a failure and you can watch the affected Activity retry while the successful executions remain completed.

And when you’re developing your next feature, you can still run:

playwright test

Nothing we’ve done requires the tests to understand Temporal. Nothing about the pattern requires Playwright or TypeScript either.

The same approach can be applied to any test suite where we can identify useful, independently executable units of work and put an Activity around the existing way of running them.

So if you’ve got a Java test suite that occasionally ruins an overnight build, a Go integration suite that takes an hour to run or thousands of Ruby specs spread across CI jobs, you don’t need to replace your test framework.

Keep the tests, the test runner, put durable orchestration around them, and stop restarting everything.