This post introduces the concept of *durable execution*, which is used by Stripe, Netflix, Coinbase, Snap, and many others to solve a wide range of problems in distributed systems. Then it shows how simple it is to write durable code using Temporal's TypeScript/JavaScript SDK.

# Distributed systems

When building a request-response monolith backed by a single database that supports transactions, we don’t have many distributed systems concerns. We can have simple failure modes and easily maintain accurate state:

- If the client can’t reach the server, the client retries.
- If the client reaches the server, but the server can’t reach the database, the server responds with an error, and the client retries.
- If the server reaches the database, but the transaction fails, the server responds with an error, and the client retries.
- If the transaction succeeds but the server goes down before responding to the client, the client retries until the server is back up, and the transaction fails the second time (assuming the transaction has some check–like an idempotency token–to tell whether the update has already been applied), and the server reports to the client that the action has already been performed.

As soon as we introduce a second place for state to live, whether that’s a service with its own database or an external API, handling failures and maintaining consistency (accuracy across all data stores) gets significantly more complex. For example, if our server has to charge a credit card and also update the database, we can no longer write simple code like:

```js
function handleRequest() {
  paymentAPI.chargeCard()
  database.insertOrder()
  return 200
}
```

If the first step (charging the card) succeeds, but the second step (adding the order to the database) fails, then the system ends up in an inconsistent state; we charged their card, but there’s no record of it in our database. To try to maintain consistency, we might have the second step retry until we can reach the database. However, it’s also possible that the process running our code will fail, in which case we’ll have no knowledge that the first step took place. To fix this, we need to do three things: 

- Persist the order details
- Persist which steps of the program we’ve completed
- Run a worker process that checks the database for incomplete orders and continues with the next step

That, along with persisting retry state and adding timeouts for each step, is a lot of code to write, and it’s easy to miss certain edge cases or failure modes (see [the full, scalable architecture](https://temporal.io/blog/workflow-engine-principles)). We could build things faster and more reliably if we didn’t have to write and debug all that code. And we don’t have to, because we can use durable execution.

# Durable execution

Durable execution systems run our code in a way that persists each step the code takes. If the process or container running the code dies, the code automatically continues running in another process with all state intact, including call stack and local variables. 

Durable execution ensures that the code is executed to completion, no matter how reliable the hardware or how long downstream services are offline. Retries and timeouts are performed automatically, and resources are freed up when the code isn’t doing anything (for example while waiting on a `sleep(‘1 month’)` statement).

Durable execution makes it trivial or unnecessary to implement distributed systems patterns like event-driven architecture, task queues, [sagas](https://microservices.io/patterns/data/saga.html), [circuit breakers](https://microservices.io/patterns/reliability/circuit-breaker.html), and [transactional outboxes](https://microservices.io/patterns/data/transactional-outbox.html). It’s programming on a higher level of abstraction, where you don’t have to be concerned about transient failures like server crashes or network issues. It opens up new possibilities like:

- Storing state in local variables instead of a database, because local variables are automatically stored for us 
- Writing code that sleeps for a month, because we don’t need to be concerned about the process that started the sleep still being there next month, or resources being tied up for the duration
- Functions that can run forever, and that we can interact with (send commands to or query data from)

Some examples of durable execution systems are Azure Durable Functions, Amazon SWF, Uber Cadence, Infinitic, and [Temporal](https://temporal.io/) (where I work). At the risk of being less than perfectly objective, I think Temporal is the best of these options 😊.

# Durable JavaScript

Now that we’ve gone over consistency in distributed systems and what durable execution is, let’s look at a practical example. I built this food delivery app to show what durable code looks like and what problems it solves:

[temporal.menu](https://temporal.menu/)

![Durable Delivery app menu](//images.ctfassets.net/0uuz8ydxyd9p/23viK26Hvfx8jYCam3kC9r/36321dc6b375fe829a8c3e701b85bb9f/menu.jpg)

*Don’t blame me for the logo—that’s just what Stable Diffusion gives you when you ask it for a durable delivery app logo. 🤷‍♂️😄*

The app has four main pieces of functionality:

- Create an order and charge the customer
- Get order status
- Mark an order picked up
- Mark an order delivered

![The order process, showing both the menu and driver sites](//images.ctfassets.net/0uuz8ydxyd9p/5t9ArgfJP8K0UGFpOAzidi/5b37dc9c647370714bd5b53b692615d0/delivery-demo.gif)

When we order an item from the menu, it appears in the delivery driver site ([drive.temporal.menu](https://drive.temporal.menu)), and the driver can mark the order as picked up, and then as delivered.

All of this functionality can be implemented in a single function of durable JavaScript or TypeScript. We’ll be using the latter—I recommend TypeScript and our library is named the TypeScript SDK, but it’s published to npm as JavaScript and can be used in any Node.js project.

## Create an order

Let’s take a look at the code for this app. We’ll see a few API routes but mostly go over each piece of the single durable function named `order`. If you’d like to run the app or view the code on your machine, this will download and set up the project:

```sh
npx @temporalio/create@latest --sample food-delivery
```

When the user clicks the order button, the React frontend calls the `createOrder` mutation defined by the tRPC backend. The `createOrder` API route handler creates the order by starting a durable `order` function. Durable functions—called *Workflows*—are started using a Client instance from `@temporalio/client`, which has been added to the tRPC context under `ctx.temporal`. The route handler receives a validated `input` (an object with a `productId` number and `orderId` string) and it calls `ctx.temporal.workflow.start` to start an `order` Workflow, providing `input.productId` as an argument:

[apps/menu/pages/api/[trpc].ts](https://github.com/temporalio/samples-typescript/blob/main/food-delivery/apps/menu/pages/api/%5Btrpc%5D.ts)

```ts
import { initTRPC } from '@trpc/server'
import { z } from 'zod'
import { taskQueue } from 'common'
import { Context } from 'common/trpc-context'
import { order } from 'workflows'

const t = initTRPC.context<Context>().create()

export const appRouter = t.router({
  createOrder: t.procedure
    .input(z.object({ productId: z.number(), orderId: z.string() }))
    .mutation(async ({ input, ctx }) => {
      await ctx.temporal.workflow.start(order, {
        workflowId: input.orderId,
        args: [input.productId],
        taskQueue,
      })

      return 'Order received and persisted!'
    }),
```

The `order` function starts out validating the input, setting up the initial state, and charging the customer:

[packages/workflows/order.ts](https://github.com/temporalio/samples-typescript/blob/main/food-delivery/packages/workflows/order.ts)

```ts
type OrderState = 'Charging card' | 'Paid' | 'Picked up' | 'Delivered' | 'Refunding'

export async function order(productId: number): Promise<void> {
  const product = getProductById(productId)
  if (!product) {
    throw ApplicationFailure.create({ message: `Product ${productId} not found` })
  }

  let state: OrderState = 'Charging card'
  let deliveredAt: Date

  try {
    await chargeCustomer(product)
  } catch (err) {
    const message = `Failed to charge customer for ${product.name}. Error: ${errorMessage(err)}`
    await sendPushNotification(message)
    throw ApplicationFailure.create({ message })
  }

  state = 'Paid'
```

Any functions that might fail are automatically retried. In this case, `chargeCustomer` and `sendPushNotification` both talk to services that might be down at the moment or might return transient error messages like “Temporarily unavailable.” Temporal will automatically retry running these functions (by default indefinitely with exponential backoff, but that’s configurable). The functions can also throw non-retryable errors like “Card declined,” in which case they won’t be retried. Instead, the error will be thrown out of `chargeCustomer(product)` and caught by the catch block; the customer receives a notification that their payment method failed, and we throw an `ApplicationFailure` to fail the `order` Workflow.
## Get order status

The next bit of code requires some background: Normal functions can’t run for a long time, because they’ll take up resources while they’re waiting for things to happen, and at some point they’ll die when we deploy new code and the old containers get shut down. Durable functions can run for an arbitrary length of time for two reasons: 

- They don’t take up resources when they’re waiting on something.
- It doesn’t matter if the process running them gets shut down, because execution will seamlessly be continued by another process. 

So although some durable functions run for a short period of time—like a successful money transfer function—some run longer—like our order function, which ends when the order is delivered, and a customer function that lasts for the lifetime of the customer.

It’s useful to be able to interact with long-running functions, so Temporal provides what we call *Signals* for sending data into the function and *Queries* for getting data out of the function. The driver site shows the status of each order by sending Queries to the order functions through this API route:

[apps/menu/pages/api/[trpc].ts](https://github.com/temporalio/samples-typescript/blob/cb617abb3e0f58a6911c66615f9c2c665e0307b0/food-delivery/apps/menu/pages/api/%5Btrpc%5D.ts#L23-L25)

```ts
  getOrderStatus: t.procedure
    .input(z.string())
    .query(({ input: orderId, ctx }) => ctx.temporal.workflow.getHandle(orderId).query(getStatusQuery)),
```

It gets a handle to the specific instance of the order function (called a *Workflow Execution*), sends the `getStatusQuery`, and returns the result. The `getStatusQuery` is defined in the order file and handled in the order function:

[packages/workflows/order.ts](https://github.com/temporalio/samples-typescript/blob/cb617abb3e0f58a6911c66615f9c2c665e0307b0/food-delivery/packages/workflows/order.ts#L55-L57)

```ts
import { defineQuery, setHandler } from '@temporalio/workflow'

export const getStatusQuery = defineQuery<OrderStatus>('getStatus')

export async function order(productId: number): Promise<void> {
  let state: OrderState = 'Charging card'
  let deliveredAt: Date

  // …

  setHandler(getStatusQuery, () => {
    return { state, deliveredAt, productId }
  })
```

When the order function receives the `getStatusQuery`, the function passed to `setHandler` is called, which returns the values of local variables. After the call to `chargeCustomer` succeeds, the state is changed to `’Paid’`, and the driver site, which has been polling `getStatusQuery`, gets the updated state. It displays the “Pick up” button. 

## Picking up an order

When the driver taps the button to mark the order as picked up, the site sends a `pickUp` mutation to the API server, which sends a `pickedUpSignal` to the order function:

[apps/driver/pages/api/[trpc].ts](https://github.com/temporalio/samples-typescript/blob/main/food-delivery/apps/driver/pages/api/%5Btrpc%5D.ts)

```ts
  pickUp: t.procedure
    .input(z.string())
    .mutation(async ({ input: orderId, ctx }) => 
      ctx.temporal.workflow.getHandle(orderId).signal(pickedUpSignal)
    ),
```

The order function handles the Signal by updating the state:

[packages/workflows/order.ts](https://github.com/temporalio/samples-typescript/blob/cb617abb3e0f58a6911c66615f9c2c665e0307b0/food-delivery/packages/workflows/order.ts#L42-L46)

```ts
export const pickedUpSignal = defineSignal('pickedUp')

export async function order(productId: number): Promise<void> {
  // …

  setHandler(pickedUpSignal, () => {
    if (state === 'Paid') {
      state = 'Picked up'
    }
  })
``` 

Meanwhile, further down in the function, after the customer was charged, the function has been waiting for the pickup to happen: 

[packages/workflows/order.ts](https://github.com/temporalio/samples-typescript/blob/cb617abb3e0f58a6911c66615f9c2c665e0307b0/food-delivery/packages/workflows/order.ts#L70)

```ts
import { condition } from '@temporalio/workflow'

export async function order(productId: number): Promise<void> {
  // …

  try {
    await chargeCustomer(product)
  } catch (err) {
    // …
  }

  state = 'Paid'

  const notPickedUpInTime = !(await condition(() => state === 'Picked up', '1 min'))
  if (notPickedUpInTime) {
    state = 'Refunding'
    await refundAndNotify(
      product,
      '⚠️ No drivers were available to pick up your order. Your payment has been refunded.'
    )
    throw ApplicationFailure.create({ message: 'Not picked up in time' })
  }
```

`await condition(() => state === 'Picked up', '1 min')` waits for up to 1 minute for the state to change to `Picked up`. If a minute goes by without it changing, it returns false, and we refund the customer. (Either we have very high standards for the speed of our chefs and delivery drivers, or we want the users of a demo app to be able to see all the failure modes 😄.)

## Delivery

Similarly, there’s a `deliveredSignal` sent by the “Deliver” button, and if the driver doesn’t complete delivery within a minute of pickup, the customer is refunded.

[packages/workflows/order.ts](https://github.com/temporalio/samples-typescript/blob/cb617abb3e0f58a6911c66615f9c2c665e0307b0/food-delivery/packages/workflows/order.ts#L82-L89)

```ts
export const deliveredSignal = defineSignal('delivered')

export async function order(productId: number): Promise<void> {
  setHandler(deliveredSignal, () => {
    if (state === 'Picked up') {
      state = 'Delivered'
      deliveredAt = new Date()
    }
  })

  // …

  await sendPushNotification('🚗 Order picked up')

  const notDeliveredInTime = !(await condition(() => state === 'Delivered', '1 min'))
  if (notDeliveredInTime) {
    state = 'Refunding'
    await refundAndNotify(product, '⚠️ Your driver was unable to deliver your order. Your payment has been refunded.')
    throw ApplicationFailure.create({ message: 'Not delivered in time' })
  }

  await sendPushNotification('✅ Order delivered!')
```

If delivery was successful, the function waits for a minute for the customer to eat their meal and asks them to rate their experience.

```ts
  await sleep('1 min') // this could also be hours or even months

  await sendPushNotification(`✍️ Rate your meal. How was the ${product.name.toLowerCase()}?`)
}
```

After the final push notification, the order function’s execution ends, and the Workflow Execution completes successfully. Even though the function has completed, we can still send Queries, since Temporal has the final state of the function saved. And we can test that by refreshing the page a minute after an order has been delivered: the `getStatusQuery` still works and “Delivered” is shown as the status:

![Poke order with Status: Delivered](//images.ctfassets.net/0uuz8ydxyd9p/2IAEpuDtPpp8NNCjeswM2h/b783df7d5c1554a50aab7a4b427b89f3/delivered.png)

# Summary

We’ve seen how a multi-step order flow can be implemented with a single durable function. The function is guaranteed to complete in the presence of failures, including:

- Temporary issues with the network, data stores, or downstream services
- The process running the function failing
- The underlying Temporal services or database going down

This addressed a number of distributed systems concerns for us, and meant that: 

- We could use local variables instead of saving state to a database.
- We didn’t need to set timers in a database for application logic like canceling an order that takes too long or for the built-in functionality of retrying and timing out transient functions like `chargeCustomer`.
- We didn’t need to set up a job queue that workers polled, either for progressing to the next step or picking up unfinished tasks that were dropped by failed processes.

In the next post, we’ll look at more of the delivery app’s code and learn how Temporal is able to provide us with durable execution. To get notified when it comes out, you can follow us on [Twitter](https://twitter.com/temporalio) or [LinkedIn](https://www.linkedin.com/company/temporal-technologies/posts/).

If you have any questions, I would be happy to help! Temporal’s mission is helping developers, and I also personally find joy in it 🤗. I’m [@lorendsr](https://twitter.com/lorendsr) on Twitter, I answer (and upvote 😄) any StackOverflow questions tagged with [`temporal-typescript`](https://stackoverflow.com/questions/ask?tags=temporal-typescript), and am @Loren on the [community Slack](https://t.mp/slack) 💃.

# Learn more

To learn more, I recommend these resources:

- Video: [Intro to Temporal and using the TypeScript SDK](https://youtu.be/6lSuDRRFgyY)
- Some [common use cases](https://temporal.io/use-cases)
- TypeScript SDK docs: [t.mp/ts](https://t.mp/ts)
- TypeScript API reference: [t.mp/ts-api](https://t.mp/ts-api)
- [TypeScript tutorials](https://learn.temporal.io/tutorials/typescript/)

More blog posts about our TypeScript SDK:

- [Using Temporal as a Node.js Task Queue](https://temporal.io/blog/using-temporal-as-a-node-task-queue)
- [Caching API Requests with Long-Lived Workflows](https://temporal.io/blog/caching-api-requests-with-long-lived-workflows)
- [Express middleware that creates a REST API for your Workflows](https://temporal.io/blog/temporal-rest)
- [1.0.0 release of the TS SDK](https://temporal.io/blog/typescript-1-0-0)
- [How we use V8 isolates to enforce Workflow determinism](https://temporal.io/blog/intro-to-isolated-vm)

*💬 Discuss on [Hacker News](https://news.ycombinator.com/item?id=34462105#34463897), [Reddit](https://www.reddit.com/r/node/comments/10hagy8/building_reliable_distributed_systems_in_nodejs/), [Twitter](https://twitter.com/lorendsr/status/1616544308886118401), or [LinkedIn](https://www.linkedin.com/posts/temporal-technologies_building-reliable-distributed-systems-in-activity-7022335275880120320-BtAg).*

*Thanks to Jessica West, Brian Hogan, Amelia Mango, and Jim Walker for reading drafts of this post.*


Building Reliable Distributed Systems in Node.js

Developer Relations Engineer

Loren Sands-Ramshaw

Loren

menu-social

#  How we built it: Building Scheduled Workflows in Temporal
Temporal’s new [Scheduler](https://docs.temporal.io/workflows#schedule) feature introduces an improved experience for developers with a schedule-specific API, finer-grained configurations, improved visibility, and the ability to have overlapping runs. For an overview of its features and use, Keith Tenzer, one of our Solutions Architects, wrote [a great blog post](https://keithtenzer.com/temporal/temporal_schedules/).

As this article is a contribution to our *How We Built It* series, I want to focus on the *Why?* and *How?* of our new Scheduler feature. As Keith covers in his post, there was a previous simple schedule feature in Temporal, but it had its drawbacks:

- Schedule couldn’t be stopped without affecting the running Workflow
- Workflow couldn’t be terminated without affecting schedule
- Schedule couldn’t be paused or have its configuration updated
- Couldn’t support overlapping runs
- Started in an unqueriable state
- No good visibility into running schedules

To improve upon these drawbacks, we knew we had a few options. 

We could build a separate scheduler system in the server using low-level components like mutable state and timer queues. This would be our most efficient option, but also the most work intensive. Ultimately, we didn’t go with this option. But, as a nod to its potential, we still designed the API with this implementation in mind, should the opportunity to switch present itself down the road. 

Another option was to build more developer-friendly functionality on top of the existing CronSchedule implementation. But, as this would mean a Schedule and a Schedule Workflow are still conflated, the problem of overlapping runs wouldn’t be solved. The inability to support overlapping runs put this option to rest.

The third, and prevailing, option was to build out the new Scheduler as a Workflow. The logic of the Scheduler process would look a lot like a Workflow even if we didn’t write it as one. And, if we did write it as one, we get to reuse a lot of functionality already in the server. Ultimately, this ability to reuse existing functionality proved a strong motivator and we moved forward with building a Workflow-based scheduler.

## Building a Workflow-based scheduler 

Deciding to build the Scheduler as a Workflow certainly wasn’t the last design choice we had to make. The next big decision we had to make was where to house the Workflow implementation. 

Do we implement the scheduler as part of the SDKs? What about as a separate library that users can link to in their workers? Or do we implement it directly on the Server workers?

SDK inclusion and shipping it as a library would mean re-implementing the scheduler Workflow in multiple languages (Go and Java don’t use our [Core SDK](https://temporal.io/blog/why-rust-powers-core-sdk)). And, in the case of implementing it as a library, users would have to worry about an additional library in their build process and potentially suffer a degraded experience from build complications.

All this considered, we decided to build the scheduler directly into Temporal Server. With this option, users wouldn’t have to install or add any additional components—they'd just make calls to an API. From a product perspective, implementing the API on the Server also gives us the flexibility to switch to a more efficient implementation later on.

### Challenges

Implementing the scheduler as a Workflow presented some unique engineering challenges. 

One of the first challenges we ran into was with Namespaces. Our existing internal Workflows run on the Worker role, in a Namespace called `temporal-system`. This is a local, not-replicated Namespace. But, our new scheduler Workflows correspond to entities (“Schedules”), which live in a user Namespace. So we asked ourselves: can we make it work if the Schedules entity lives in a user Namespace, but the Workflow that implements it lives in a system Namespace? 

We decided against trying to make this fit, primarily because of concerns around replication and migration. We support replicating a user Namespace to another Cluster and migrating active copies to new Clusters—as I mentioned above, the system Namespace is not replicated, so this would require building a lot of new replication functionality. But, if we were to run the scheduler Workflows in the user Namespace, we pretty much get replication for free.

So what did we do? 

We run the scheduler Workflows in the user Namespace. But what about the SDK worker in the Worker role? It only listens on a task queue in the system namespace. For now, we did the simplest thing possible: we just run more SDK workers! For each user Namespace, we run at least one SDK worker on one of the server Workers. There’s a new component that keeps the running workers in sync with registered Namespaces.

This may sound excessive but the SDK worker is relatively light, so hundreds of them in one process is not a problem. In the future, we plan to optimize this by allowing one SDK worker to handle multiple Namespaces and task queues at once (this may be the subject of a future post).

#### Hiding

As things stood, with the scheduler Workflows in the user Namespace, if a user were to list all Workflows, they would see these scheduler Workflows as well. We consider these an internal implementation detail and didn’t want them coming up in user queries. So we had to figure out how to “hide” these scheduler Workflows from user interactions. 

One option would be to simply exclude these Workflows from visibility. However, we need visibility to implement “list schedules”. We could push them to a separate visibility index, but our current visibility implementation only works with a single index, so this would require a lot of work. 

Then we thought, maybe we can just filter them out of the results of visibility queries that we get back from the database? But this led to pagination woes: a request for a page of 100 user Workflows might return only 95, or worse, 0, if only scheduler Workflows were found. But we were on the right track.

In the end, we decided to use a simple trick: a search attribute. We marked these Workflows with a specific search attribute, and added a clause on that attribute to all queries (unless a query mentions it specifically). To keep things general, we called this new level of grouping within a Namespace a "Namespace division." 

Seeing some benefits, we might later extend this Namespace division functionality to other elements of Temporal.

#### Signal vs. query

I noted above that two of the big improvements with this new scheduler is that it is stateful and can now be interacted with via a synchronous [gRPC](https://grpc.io/) API. So, under the hood, calls like `create schedule` and `delete schedule` are really `start workflow` and `terminate workflow`. What about `pause schedule`? 

Since commands like `pause` and `update` are changing the state of the Workflow to the effect that its behavior changes, they are implemented with **signals**. But, what about something like `describe schedule`? This got a little tricky. Part of the output of `describe schedule` is a set of the currently-running Workflows that were started by the scheduler. The scheduler knows what Workflows it started, but does it know which are still running? For the scheduler’s state to be updated, it would need to be notified when the Workflow is complete. But, the only things that can wake up a Workflow are a timer (or timeout), a signal, an Activity completion, or a Child Workflow completion.

The Workflows that are started by the scheduler are not Child Workflows; they're independent (this is because the scheduler Workflow is an implementation detail that we don’t want to expose and because Child Workflow completion notification doesn’t work across continue-as-new anyway). This means that the only reasonable way to get notified if a scheduled Workflow completes is to have an Activity that listens and returns when the Workflow completes. 

So based on this, we can query the Workflow for "describe", but the set of running Workflows it returns might include ones that have completed already but whose state has not yet updated. How did we avoid this?

Eventually, we’ll use a new feature that allows synchronous Workflow updates, but this is still in development. So for now, the “describe” implementation in the frontend service checks that all the “running Workflows” are still running. If any are completed, it sends the scheduler a signal, to tell it to wake up and refresh the state of all its running Workflows, and then it does the query again.

## Conclusion

Building our new scheduler as a Workflow was a great challenge and ultimately allowed us to build a feature that I really believe is going to improve the developer experience for Temporal users. 

If you want to learn more about using the new scheduler in your applications, [check out our docs](https://docs.temporal.io/) for SDK-specific information. 


clem-onojeghuo-T7gi cyrkdg-unsplash

Temporal’s new Scheduler feature introduces an improved experience for developers with a schedule-specific API, and the ability to have overlapping runs.

How We Build It: Building Scheduled Workflows in Temporal

Engineering

David Reiss

David-Reiss

Imagine you're at a café trying to order a sandwich. Most of the time everything goes fine and you get a delicious meal. And sometimes, things don't go so well. Maybe they don't make your favorite sandwich, or maybe they've run out of cheese, or maybe you've shown up right at the beginning of the lunch rush.

Some of these failures are easier to deal with than others. Some of them you have to deal with (change your order, wait for a while, or decide it's too long and leave). Others the restaurant will need to deal with (bring in more line cooks, grab the backup cheese from the fridge, or go buy more). 

It's no different for software, and with Temporal, we aim to make handling many types of failure easy.

My colleague Dominik Tornow recently [outlined the theoretical and formal basis](https://dominik-tornow.medium.com/handling-failures-from-first-principles-1ed976b1b869) of failures and what to do about them. Fundamentally, he points out, failure has two dimensions: **_where_** it happens and **_when_**. Both dimensions have an element of **_what_**: Retry the failure or not? Compensate or not? But also **_how_**? How do we handle failures once we know how to classify them?

Let's consider a process for handling an online order with three major steps:

1. Validate inventory (and possibly reserve the item(s))
1. Charge customer
1. Fulfill order

Dominik focused on a couple of potential failures that might crop up in step two (charging the customer's credit card). I'll focus on the same, but in the context of a Temporal Workflow. (For the rest of this post, I'll assume you have at least a passing familiarity with Temporal's core concepts.)

With Temporal, you'd implement this three step process using a Temporal Workflow that executes three separate Activities. Here's an example of this Workflow in Go:

```go
package app

import (
	"time"
	"go.temporal.io/sdk/workflow"
)

type OrderInfo struct { /* ... */ }

func Workflow(ctx workflow.Context, order OrderInfo) error {
	activityoptions := workflow.ActivityOptions{
		StartToCloseTimeout: 5 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, activityoptions)

	var a *Activities

	err := workflow.ExecuteActivity(ctx, a.CheckInventory, order).Get(ctx, nil)
	if err != nil {
		return err
	}

	err = workflow.ExecuteActivity(ctx, a.Charge, order).Get(ctx, nil)
	if err != nil {
		return err
	}

	err = workflow.ExecuteActivity(ctx, a.FulfillOrder, order).Get(ctx, nil)
	if err != nil {
		return err
	}

	return nil
}
```

That is, one activity to represent each of the major steps. But, of course, failures occur! What happens if one of these stages fails? Is that `StartToCloseTimeout` hanging out at the top of the function implying a retry after five seconds? Surely that doesn't cover everything that could possibly go wrong... What ever will we do?!?

## Where: Platform vs. Application

In the "where" dimension, what Dominik termed _spatial_, the `Charge` activity has two potential errors: `InsufficientFundsError` and `ConnectionError`.

One way to think about how to classify these is to ask yourself: Can the error, in most cases, be handled entirely by something low-level? (Think: TCP retransmission.) If yes, then it's most likely a platform-level failure.

From another perspective, you could ask yourself: Does handling the error require scenario or domain-specific logic? (Think: changing your sandwich order; the restaurant, aka platform, doesn't know what you want.) If yes, then it's most likely an application-level failure.

For the two example errors, nothing about the request itself—not which account we're charging, or how much money to charge—would stop `ConnectionError` from happening. That makes it platform-level. In Temporal, you can configure the platform to handle errors like this automatically:

```go
	retrypolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     1.0,
		NonRetryableErrorTypes: []string{"InsufficientFundsError"},
	}

	activityoptions := workflow.ActivityOptions{
		RetryPolicy: retrypolicy,
	}
	ctx = workflow.WithActivityOptions(ctx, activityoptions)
	err = workflow.ExecuteActivity(ctx,
		a.Charge,
		order.Id,
		order.TotalCost).Get(ctx, nil)
```

But don't do this! A linear retry policy like this isn't great. I'll be working up to an exponential backoff in the next section.

After the first failure, the `InitialInterval` tells Temporal to wait one second before retrying. If the `Charge` activity fails again, the wait time will be that interval multiplied by the `BackoffCoefficient` for every subsequent attempt. As such, given retry attempt number `N`, [we'll calculate the delay](https://github.com/temporalio/temporal/blob/db09e2801fb7b2e5a01e579a1a2fa2b8cd31f795/service/history/workflow/retry.go#L96) to be:

![Backoff Equation](//images.ctfassets.net/0uuz8ydxyd9p/6hnDC6DAMQRFOzEXOnscq4/73ed39c9b8e8334473bd02ff29a299ab/Screenshot_2022-11-03_at_5.31.13_PM.png)

(That is, `InitialInterval` times `BackoffCoefficient` raised to the `N`th power.)

So, with both the coefficient and the initial interval set to 1.0, this tells Temporal to retry a failed activity every second. This works great for most cases of `ConnectionError`. If there's a network blip, chances are good that the blip will be done after a single second.

I've listed `InsufficientFundsError` in `NonRetryableErrorTypes` because it's a totally different story. What would happen if that error was retried after a second? What are the chances the customer's account actually has enough money now? Pretty low unless the activity execution caught them on payday at the exact moment the check was being deposited.

That makes `InsufficientFundsError` an application-level failure. Retrying it almost certainly won't make things better. You'll have to resort to business logic to handle this error, likely backtracking to display an error message to the customer and if the item(s) were reserved in inventory, release them back to the available pool.

Additionally, because the strategy for handling `InsufficientFunds` errors involves undoing some work, you'd say that it's _Backward Recovery_.

The handling strategy I've used here for `ConnectionError` is, well, _mostly_ Forward Recovery, because if a retry succeeds, we can just continue on without undoing anything. But it's also ambiguous.

## When: temporal (but not **_T_**emporal) failures

Does retrying after a `ConnectionError` every second for_ eternity_ make sense? Like all things, it depends. (Spoiler alert: it probably doesn't make sense.)

[Dominik explained](https://dominik-tornow.medium.com/handling-failures-from-first-principles-1ed976b1b869) three kinds of failures along the **_when_** dimension (that is, their characteristics change over time): Transient, Intermittent, and Permanent.

**Transient** errors are almost the easiest to deal with because they tend to go away quickly. Momentary network congestion, a router restarting, a new server version rolling out mid-request... These will resolve themselves and so the Retry Policy I wrote above, with no backoff, will be fine.

But if you consider that `1.0` constant, you'll end up in the realm of **Intermittent** failures pretty quickly. Like transient errors, intermittent errors will likely resolve themselves, but it's going to take a moment; immediately retrying is probably going to give you the same error. Think rate limits, power outages, unplugged network cables. The fix? Actually back off on those retries and retry them with ever increasing waits:

```go
	retrypolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     2.0,
		NonRetryableErrorTypes: []string{"InsufficientFundsError"},
	}
```

Recall the retry formula above. By setting `BackoffCoefficient` to something greater than 1, you've now got an exponential backoff on the retries with no additional code. 

Transient and Intermittent failures are closely related. Generally speaking, you can expect that a retry will eventually succeed. When it does, this would be a _Forward Recovery_.

The line between those and a **Permanent** failure is often hard to find. In some ways, the application-level error, `InsufficientFundsError`, is a permanent error because without intervention – someone depositing money in the account or switching to a different card – you can't expect this error will resolve itself.

Likewise, `ConnectionError` can be permanent. In some cases, you might be able to detect these sub-errors explicitly – authentication failure, 404. If that's the case, just add them to the list of non-retryable types.

But trying to exhaustively list all known permanent, non-retryable errors would be exhausting. Instead, you have two mitigation strategies you can use.

First, set a couple of maximums on the retry policy, so that no matter what, the workflow won't just sit there spinning on something that'll never work. (Depending on the API, this could be expensive: retrying a call to [Google's Nearby Places API](https://developers.google.com/maps/documentation/places/web-service/usage-and-billing#nearby-search) every second for an hour, for example, would cost $115.20.)

```go
	retrypolicy := &temporal.RetryPolicy{
		InitialInterval:        time.Second,
		BackoffCoefficient:     2.0,
		MaximumInterval:        time.Second * 120,
		MaximumAttempts:        50,
		NonRetryableErrorTypes: []string{"InsufficientFundsError"},
	}
```

Then, consider modifying the activity itself to use [`NewNonRetryableApplicationError`](https://pkg.go.dev/go.temporal.io/temporal#NewNonRetryableApplicationError) from the Temporal SDK. Roughly:

```go
func Workflow(ctx workflow.Context, order OrderInfo) error {
	/* ... */

	retrypolicy := &temporal.RetryPolicy{
		// ...
		NonRetryableErrorTypes: []string{"InsufficientFundsError", "AuthFailure"},
	}

	/* ... */
}

func Charge(ctx context.Context, order OrderInfo) error {
	/* ... attempt to charge the credit card ... */

	var connErr *ConnectionError
	if err != nil && errors.As(err, &connErr) {
		// Without manual intervention, a 403 - Forbidden is going to keep happening. So
		// instead of returning the plain ConnectionError (which by default would be retried)
		// we'll wrap it in a NonRetryable to prevent that from needlessly happening.
		if connErr.code == 403 {
			return temporal.NewNonRetryableApplicationError(
				connErr.message,
				"ConnectionError",
				err,
			)
		}
	}

	// May be nil or not. If not, let the caller (i.e., workflow) decide if
	// it's retryable by listing non-retryables in the retry policy and handling appropriately.
	return err
}
```

From there, if the workflow receives one of those non-retryable errors or exceeds the retry attempts you'll probably want to "recover backward" and undo anything critical that the process had committed (such as reserving inventory that the customer can't actually pay for).

In pseudocode, such a rewrite of the workflow might look like the following:

```go
const (
	STAGE_INVENTORY = 0
	STAGE_CHARGE    = 1
	STAGE_FULFILL   = 2
)

err := ExecuteActivity(CheckInventory)
if err != nil {
	recover(STAGE_INVENTORY)
	return err
}

err = ExecuteActivity(Charge)
if err != nil {
	recover(STAGE_CHARGE)
	return err
}

err = ExecuteActivity(FulfillOrder)
if err != nil {
	recover(STAGE_FULFILL)
	return err
}

func recover(stage int) {
	switch stage {
	default:
		// For unknown stages of the process, attempt to undo the whole thing.
		fallthrough
	case STAGE_FULFILL:
		// Kick off undoing the entire order
		ExecuteActivity(ReturnToWarehouse)
		fallthrough
	case STAGE_CHARGE:
		// Given a transaction ID, ensure the user isn't charged.
		ExecuteActivity(ReverseCharge)
		fallthrough
	case STAGE_INVENTORY:
		ExecuteActivity(ReleaseInventory)
		fallthrough
	case -1:
		ExecuteActivity(NotifyCustomerFailure)
	}
}
```

## Wrapping up

Errors and failures don't have to be scary. Thinking about where the error occurs and how long it might last will give you a strong framework for deciding what to do about it—either failing forward (retry until success) or backtracking (cease trying, undo actions as necessary, and report failure).

Temporal helps you deal easily and seamlessly with a wide range of failure modes. You'll still have to figure out how to nicely tell the user your API key expired, but at least now you'll _know_ you handled it the best you could.

If you want to see the complete code used in this post, including some simulations of the various failure scenarios, it's up in [this GitHub repo](https://github.com/afitz0/firstfailure).

social-waves

Developer Advocate

Fitz

Failure Handling in Practice

I recently had the honor of teaching Temporal 101 at our inaugural [Replay conference](https://temporal.io/replay "Replay conference") in Seattle. With more than 100 people attending that session, it was one of the largest live hands-on training classes I've ever taught. For every developer who had the opportunity to participate that afternoon, I imagine that at least 10,000 could not. 

Fortunately, you won't have to wait for Replay 2023 or even book your flight to Seattle to learn Temporal. We recently published an online, self-paced version of the [Temporal 101 course](https://temporal.talentlms.com/catalog/info/id:126 "Temporal 101 course"). It's ideal for developers who are new to Temporal and want to quickly learn the fundamentals. 

Best of all, this course assumes no prior knowledge of Temporal and is completely free. Registration is required, however, so that you can save your progress and we can verify course completions. The course combines written content, videos (with complete transcripts), assessments, and hands-on exercises. We find that most learners can complete the course in about two hours, but you can go as slow or as fast as you like. 
![blog-Temporal 101-1](//images.ctfassets.net/0uuz8ydxyd9p/4mTo83itg6zjkQQT8gjWEz/2c8191df7efb32a8ff38c80795ae1578/blog-1.png)

### Little To No Experience Needed
Although the code examples and hands-on exercises are written in Go, they are presented with explanations and avoid advanced Go language features. Learners have reported success with the course despite having little or no experience with Go.
![blog-Temporal 101-2](//images.ctfassets.net/0uuz8ydxyd9p/77hYlMRHDRwifgZSHSqyqZ/5b1145ed5f6203dc2311b7ebbe1d0c47/blog-2.png)

### It's Easily Accessible
I understand that not everyone is able or willing to install software on their computers. That's why this course uses a browser-based exercise environment provided through [GitPod](https://gitpod.io "Gitpod"), available at no cost to you. Using a standard Web browser such as Chrome or Firefox, you'll have access to a private Temporal Cluster, as well as terminal windows where you can type commands and an editor you can use to write your code. The only thing you'll need besides a Web browser is a [GitHub](https://github.com "GitHub") account that you can log into from that browser. (GitPod uses your account to install the exercise code you use during the course.)
![blog-Temporal 101-3](//images.ctfassets.net/0uuz8ydxyd9p/3366rSrzeYuidLHsqZJ6VR/cca9e566fd57d860894cca343b0d3cac/blog-3.png)

We announced this course more than a week ago to members of our Temporal Education Updates mailing list. If you'd like to be among the first to learn about new courses, including the availability of *Temporal 101* for other programming languages, I encourage you to [sign up](https://pages.temporal.io/get-updates-education "sign up") so we can notify you, too.

We recently published an online, self-paced version of the Temporal 101 course - ideal for developers who want to quickly learn the fundamentals.

Self-Paced Temporal 101 Training: Learn Anywhere and Anytime

Principal Curriculum Developer

Tom Wheeler

Free Self-Paced Temporal 101 Training: Learn Anywhere and Anytime

*This blog post is a public copy of our monthly newsletter sent on July 29. If you'd like to receive upcoming updates by email, you can [subscribe to our newsletter here](https://pages.temporal.io/newsletter-subscribe "Subscribe to the Temporal newsletter").* 

This month's newsletter features some exciting news and updates, including:

*   Our Typescript SDK v1.0.0 is now available! [Read more in this blog post ›](https://docs.temporal.io/blog/typescript-1.0.0)
*   [Replay](https://temporal.io/replay), Temporal's first-ever developer experience conference, is next month! Join us on August 26 in Seattle for talks from Temporal users at companies like Stripe, Datadog, Snap Inc, and Yum! Brands.

For more information on what we've been working on, just keep reading! And as always, we'd love to hear from you—feel free to share feedback in our [Community Slack group](https://temporal.io/slack) or on Twitter ([@temporalio](https://twitter.com/temporalio)).

Technology
----------

### Developer Experience

**TypeScript GA**: We are incredibly excited to announce that our TypeScript SDK is officially GA. This SDK has been a labor of love and a long time coming. It boasts an absolutely killer developer experience, powered by V8 isolation technology. We believe the TypeScript SDK represents the fundamental direction all of our future SDKs will go.

[Typescript SDK ›](https://github.com/temporalio/sdk-typescript)

[Typescript SDK test framework ›](https://docs.temporal.io/typescript/testing/)

**Web UI V2 GA:** Although the core technology of Temporal is a true game-changer, capitalizing on that value would be nearly impossible without high quality tools to support it. A year ago we took a hard look at our existing UI experience and realized it was holding the overall product back. Since then, we hired amazing Frontend and Design teams who have diligently worked to match our UI quality to what you've come to expect from Temporal. We are incredibly excited to announce that the experience is now GA! If you haven't already made the switch, we highly encourage it.

As part of this release, we are also planning to deprecate our previous Web UI. The timeline for that deprecation is **September 30, 2022**. That means you have a little over two months to switch to the new Temporal UI.

**Python Alpha:** Earlier this year we started developing a [Python SDK](https://github.com/temporalio/sdk-python) for Temporal. The team has been hard at work and immense progress has been made. It will still take a while before we can seriously target a GA release but the Alpha client is surprisingly full featured. We want to be incredibly clear that any Alpha/Beta release should not be used for critical workloads. That being said, we can never get too much community feedback about the SDK experience.

**Spring Boot Alpha support:** For those who are embedded in the Java ecosystem, developing applications without Spring Boot is almost absurd. That's why it's a little strange that Temporal's Java SDK has never had official bindings for the Spring ecosystem. We're serious about changing that, and this Alpha release is the first step in the right direction. [PR #1305 ›](https://github.com/temporalio/sdk-java/pull/1305)

**tctl V2 GA:** In the developer world there are really two groups, "UI people" and "command line people." We want to make sure the experience is equally great regardless of your tooling preference, so the team has been hard at work revamping the Temporal command-line tool `tctl`. The goal was to keep the experience familiar while removing the inconsistent developer experience, rough edges, and missing functionality. `tctl` V2 is now GA and will become the default in the next release.

As part of this release, we are also planning to deprecate `tctl` V1. The timeline for that deprecation is **September 30, 2022**. That means you have a little over two months to switch to the new `tctl`.

**Schedules Beta:** Cron jobs are everywhere and running them may seem simple on the surface, doing so reliably is much easier said than done. Fortunately, Temporal is designed for running time based workloads reliably at any scale. Unfortunately, our current cron experience leaves…well, a lot to be desired. The functionality is limited, the experience is confusing, and debugging things when they go wrong is seriously unpleasant. Due to these shortcomings, the team has poured their hearts into a fundamentally new experience. The improvements are so significant that we are even renaming the feature to Schedules. Schedules offer all the capabilities of the original cron functionality, but are also:

*   Easy to use
*   Straightforward to debug
*   Functionally rich
*   Fully extensible

We don't have a concrete timeline for GA of this feature, but we need **your** feedback. We encourage you to grab the latest server and tctl release and give the new experience a spin.

### See also

*   Eager activity dispatch for Python, TypeScript, and Go
*   Started development of a [Ruby SDK](https://github.com/temporalio/sdk-ruby)
*   [Local activities](https://docs.temporal.io/php/activities#local-activity) are available in the PHP SDK
*   Go SDK [v1.15.0](https://github.com/temporalio/sdk-go/releases/tag/v1.15.0) released
*   Java SDK [v1.14.0](https://github.com/temporalio/sdk-java/releases/tag/v1.14.0) released
*   Workflow deletion is available as a `tctl` command that does not require direct DB access from `tctl`
*   Namespace deletion is also now available as a `tctl` command
*   Host-level priority Task processing

Content and Community
---------------------

**Support team growth:** As many of you know, we take support seriously at Temporal. Historically the support workload has been carried by members of various engineering and product teams within the company. Recently, we’ve formalized this area of work and established an official support team. Until last month Tihomir (whom many of you recognize as your go-to resource for support questions, was our sole member of this team. We are incredibly excited to welcome two new support team members Antonio Mendoza and Jordan Reynolds. Antonio and Jordan both bring an intense passion for technology and helping others. They will be key in scaling the response and high-quality support we provide.

**Community knowledge base:** Temporal offers a few options when it comes to learning more or getting your questions answered. We have our support forum for standard Q&A, Slack for community engagement, and docs for establishing foundational product knowledge. In general we are very happy with these solutions, but recently we felt that some classes of questions that aren’t being well addressed. In an attempt to solve for this class of questions, we are experimenting with adding a knowledge base to our community support forum ([community.temporal.io](https://community.temporal.io/)). Our hope is that this knowledge base will surface a set of information that is not covered by the docs or easily discoverable on the support forums. We are especially interested in hearing your feedback on this one!

**Python in the Application development guide:** A few months ago we released a new style of product documentation that aimed to provide a cohesive narrative instead of “requiring assembly.” So far this new approach has been well received and therefore we are confident to continue in this direction. Considering the immense progress we have made in developing the Python SDK, adding Python to the app dev guide seemed like a no-brainer. The content isn’t fully complete, but most of the Foundational section has been updated to include Python at this point. [Python in the Application development guide ›](https://docs.temporal.io/application-development/?lang=python)

**Observability section of Application development guide:** In addition to supporting new languages for existing App dev guide content, we are also adding net new information to help guide your development journey. The guide now has a section about [observability](https://docs.temporal.io/application-development/observability/), which continues to be one of the most requested topic areas in Temporal.

**New home for tutorials:** Historically all learning content has lived on [docs.temporal.io](https://docs.temporal.io). But with the founding of our Education team earlier this year, the landscape is changing. Moving forward, learning content will be available over at [learn.temporal.io](https://learn.temporal.io), and the majority of our existing learning content has already been migrated. For now, the migrated content makes up the entirety of the learning site, but that won’t be the case for long. Stay tuned for some very exciting educational updates coming from the team!

### See also

*   Check out our new [Community Ask and Learn](https://www.youtube.com/playlist?list=PLl9kRkvFJrlRh_OrohmwIvE8OsXSsi7QV) video series
*   [Temporal Cloud release notes](https://docs.temporal.io/cloud/release-notes/) are now available
*   [Experimental Schedule feature documentation](https://docs.temporal.io/workflows#schedules) is now available
*   Last call for feedback on our [quarterly docs survey](https://forms.gle/VWgQdPbV5qa8zJVBA)
*   \[Video\] [Temporal Explained: Idempotence](https://www.youtube.com/watch?v=JpBNKuYMB10)
*   \[Video\] [Temporal Flagship Community Meetup - June 2022](https://www.youtube.com/watch?v=yPZK82Kwe3o)

Community Spotlight
-------------------

Very few things that make us more excited than seeing community-created Temporal content, which is why we're so excited about this:

[Write your first Temporal workflow in Typescript ›](https://www.bitovi.com/blog/write-your-first-temporal-workflow-in-typescript)

We missed this when it was originally published, but fortunately it was recently shared with the team and it made our collective day. In this post about developing Workflows in TypeScript, Michael Haynie does an outstanding job of presenting a practical and digestible path for getting started with Temporal. If you've been waiting for that extra push to get started with our now-GA TypeScript SDK, wait no longer. Posts like these are what make our community so great and we would love to see more! The team is always happy to support content creators, so if you're looking for technical feedback, or a spelling check, or you're just feeling a bit anxious about putting something out there, you're not alone.

### Twitter thread of the month

[![Dominik tweet: Are compensating actions a defining component of sagas or a frequent but non-defining component of sagas?! Well ............ What even are compensating actions?!?!](https://pages.temporal.io/rs/250-WIU-007/images/dominik-tornow-saga-showdown.png)](https://twitter.com/DominikTornow/status/1547685647975997440)

Check out [@DominikTornow](https://twitter.com/DominikTornow/status/1547685647975997440) on "The Saga Showdown."


mo-U3Kst7MY4Ok-unsplash

Temporal Community Newsletter: July 2022

Charles Zedlewski

charles zedlewski


There are numerous task queues for Node.js. [Bull](https://optimalbits.github.io/bull/) is the most popular, but [Bee Queue](https://www.npmjs.com/package/bee-queue), [Agenda](https://www.npmjs.com/package/agenda), and [Kue](https://www.npmjs.com/package/kue) are also common.

All these task queue libraries solve a similar problem. They let you put a task onto a queue and define a worker that consumes tasks from the queue. While Temporal is much more than a task queue, Temporal does solve the same problems as these task queue libraries. Plus, Temporal's feature set provides several advantages over conventional task queues.

## Setting Up a Task Queue With Temporal

A common use case for a task queue is sending a welcome email when a new user registers. Sending email is a good candidate for a task queue because sending an email often requires a slow API call and is more of a "side effect" of registering the user. The following is a basic example of defining an email task queue with Bull. If you're interested in a more complete tutorial, here's a [tutorial about building an email queue with Bull in TypeScript](https://blog.taskforce.sh/implementing-mail-microservice-with-bullmq/).

`email-queue.js`

```javascript
const Queue = require('bull');
const axios = require('axios');

const emailQueue = new Queue('email');
const mailgunAPI = 'https://api.mailgun.net/v3';

emailQueue.process(async function sendEmail(job) {
  const { to, from, subject, html } = job.data;
  await axios({
    url: `${mailgunAPI}/${domain}/messages`,
    method: 'post',
    params: { to, from, subject, html },
    auth: {
      username: 'api',
      password: apiKey
    }
  })
});
```

`create-test-task.js`

```javascript
const Queue = require('bull');

const emailQueue = new Queue('email');

emailQueue.add({
  to: 'john.smith@gmail.com',
  from: 'test@temporal.io',
  subject: 'Welcome to Temporal!',
  html: 'This is a test email'
});
```

With Temporal, the setup is similar. A task or job corresponds to a [Workflow](https://docs.temporal.io/concepts/what-is-a-workflow-execution). But, first, you need to define an [Activity](https://docs.temporal.io/concepts/what-is-an-activity) to send the email as follows.

`src/activities.ts`

```ts
import axios from 'axios';
import { SendEmailParams } from './interfaces';

interface MailgunSettings {
  apiKey: string;
  domain: string;
}

const mailgunAPI = 'https://api.mailgun.net/v3';

export const createActivities = ({ apiKey, domain }: MailgunSettings) => ({
  async sendEmail(params: SendEmailParams): Promise<void> {
    const { to, from, subject, html } = params;
    await axios({
      url: `${mailgunAPI}/${domain}/messages`,
      method: 'post',
      params: { to, from, subject, html },
      auth: {
        username: 'api',
        password: apiKey,
      },
    });
  },
});
```

Next, you need to define a Workflow that calls this activity.

`src/workflows.ts`

```ts
import { proxyActivities } from '@temporalio/workflow';
import type { createActivities } from './activities';
import { SendEmailParams } from './interfaces';

const { sendEmail } = proxyActivities<ReturnType<typeof createActivities>>({
  startToCloseTimeout: '5m',
});

export async function sendEmailWorkflow(params: SendEmailParams): Promise<void> {
  await sendEmail(params);
}
```

Finally, you need to create a [Worker](https://docs.temporal.io/concepts/what-is-a-worker-entity) to process tasks as follows.

`src/worker.ts`

```ts
import { Worker } from '@temporalio/worker';
import dotenv from 'dotenv';
import { createActivities } from './activities';

dotenv.config();

const {
  MAILGUN_API: apiKey,
  MAILGUN_DOMAIN: domain
} = process.env;

async function run(): Promise<void> {
  if (apiKey === undefined) throw new Error('No API key');
  if (domain === undefined) throw new Error('No domain');

  const activities = createActivities({
    apiKey,
    domain,
  });

  const worker = await Worker.create({
    taskQueue: 'email',
    activities,
    workflowsPath: require.resolve('./workflows'),
  });
  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Finally, to start a Workflow, you need to create a `Client` and `start()` the Workflow as follows.

`src/client.ts`

```ts
import { Client } from '@temporalio/client';
import { sendEmailWorkflow } from './workflows';
import { nanoid } from 'nanoid';

const client = new Client();

const handle = await client.workflow.start(sendEmailWorkflow, {
  args: [{
    to: 'john.smith@gmail.com',
    from: 'test@temporal.io',
    subject: 'Welcome to Temporal!',
    html: 'This is a test email'
  }],
  taskQueue: 'email',
  workflowId: 'workflow-' + nanoid(),
});
```

The Temporal code is slight more verbose because you need a separate Activity function. However, Temporal has several features that make it an excellent option for the task queue use case.

## Additional Features: UI, Type Checking, Retries, Delays

Most Node.js task queues have a corresponding UI for monitoring and managing tasks. For example, [Taskforce](https://taskforce.sh/) for Bull, or [Agendash](https://github.com/agenda/agendash) for Agenda. The Temporal UI lets you manage and monitor Workflows in a similar way.

The Temporal UI for an execution of `sendEmailWorkflow()` looks like the following. It displays when the `sendEmailWorkflow()` started and ended, when the `sendEmail()` activity started and ended, and the data `sendEmail()` returned.

![Screenshot of a sendEmailWorkflow in the UI](//images.ctfassets.net/0uuz8ydxyd9p/45T3Rp0tzEkuSYctIe6XUx/5c722448a042f2bc336bbde0b8d19a0a/ui-send-email.png)

Temporal has stronger built-in type safety than conventional task queues. For example, Bull doesn't provide a way for TypeScript to tie the `email` queue to the arguments that the `sendEmail()` handler expects. You can call `emailQueue.add(args)` with an arbitrarily typed argument. On the other hand, TypeScript type checks `args` in the following code to make sure it lines up with the parameters `sendEmailWorkflow()` expects.

```ts
const handle = await client.workflow.start(sendEmailWorkflow, {
  args: [{
    to: 'john.smith@gmail.com',
    from: 'test@temporal.io',
    subject: 'Welcome to Temporal!',
    html: 'This is a test email'
  }], // TypeScript can type check `args`
  taskQueue: 'email',
  workflowId: 'workflow-' + nanoid(),
});
```

Suppose the Mailgun API has a temporary issue. Ideally, your task queue would retry sending the email automatically a fixed number of times before marking the task as failed. Some task queues support retrying failed jobs: Bull does not, but [Bee Queue does](https://www.npmjs.com/package/bee-queue#retrying).

Temporal supports retrying activities with exponential backoff via [retry policies](https://docs.temporal.io/concepts/what-is-a-retry-policy/). For example, the following code shows how you might configure Temporal to retry the `sendEmail()` activity at most 2 times.

`src/workflows.ts`

```ts
import { proxyActivities } from '@temporalio/workflow';
import type { createActivities } from './activities';
import { SendEmailParams } from './interfaces';

interface SendEmailParams {
  to: string;
  from: string;
  subject: string;
  html: string;
}

const { sendEmail } = proxyActivities<ReturnType<typeof createActivities>>({
  retry: {
    maximumAttempts: 2 // Retry `sendEmail()` twice before failing
  },
  startToCloseTimeout: '5m',
});

export async function sendEmailWorkflow(params: SendEmailParams): Promise<void> {
  await sendEmail(params);
}
```

Suppose you want to delay sending your email for 24 hours, because you want to remind the user of something. Many task queues also support delaying processing a task. For example, [Bull lets you specify a `delay` in milliseconds](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd) when adding a task to a queue.

Temporal Workflows are durable, which means you can safely put a `sleep()` in your Workflow to add a delay.

`src/workflows.ts`

```ts
import { proxyActivities, sleep } from '@temporalio/workflow';
import type { createActivities } from './activities';
import { SendEmailParams } from './interfaces';

const { sendEmail } = proxyActivities<ReturnType<typeof createActivities>>({
  startToCloseTimeout: '5m',
});

export async function sendEmailWorkflow(params: SendEmailParams): Promise<void> {
  const delay = params.delay;
  if (delay !== undefined) {
    await sleep(delay);
  }

  await sendEmail(params);
}
```

The ability to use `sleep()` in your Workflows without sacrificing durability means you can write multiple "tasks" in one Workflow. If you want to send another email after a certain period of time, you can add it to your single Workflow rather than scheduling multiple tasks in a traditional task queue. Let's take a look at how you can use Workflows to represent a more sophisticated series of tasks.

## Multi-Part Tasks

Many modern apps don't just send one welcome email. They often have a whole series of emails over several days that help new customers get acquainted with the app. With a traditional task queue, you would need a separate task for every email in the series for every customer. Add in conditional logic, like sending a different email if a customer has no interactions with the app, and implementing a multi-part email series can be an extremely complex endeavor.

With Temporal, you can send multiple emails from one Workflow. In the following code, the `customerWelcomeSeriesWorkflow()` sends multiple emails separated by 24 hours.

`src/workflows.ts`

```ts
import { proxyActivities, sleep } from '@temporalio/workflow';
import type { createActivities } from './activities';

interface CustomerWelcomeSeriesParams {
  to: string;
  from: string;
}

const { sendEmail } = proxyActivities<ReturnType<typeof createActivities>>({
  retry: {
    maximumAttempts: 2
  },
  startToCloseTimeout: '5m',
});

export async function customerWelcomeSeriesWorkflow(params: CustomerWelcomeSeriesParams): Promise<void> {
  const { from, to } = params;
  await sendEmail({
    to,
    from,
    subject: 'Welcome to Temporal!',
    html: 'This is a test email'
  });

  await sleep('1 day'); // Wait for 24 hours

  await sendEmail({
    to,
    from,
    subject: 'Your First Day Using Temporal',
    html: 'This is a test email'
  });

  await sleep('1 day'); // Wait for 24 hours

  await sendEmail({
    to,
    from,
    subject: 'Advanced Temporal Workflows',
    html: 'This is a test email'
  });
}
```

You can also conditionally schedule emails based on user actions using [Signals](https://docs.temporal.io/concepts/what-is-a-signal/). For example, your app can send a Signal to `customerWelcomeSeriesWorkflow()` when a user logs in, so `customerWelcomeSeriesWorkflow()` can send a different email if a new customer hasn't logged in for 24 hours.

`src/workflows.ts`

```ts
import { defineSignal, proxyActivities, sleep } from '@temporalio/workflow';
import type { createActivities } from './activities';

export const loggedInSignal = defineSignal('loggedIn');

interface CustomerWelcomeSeriesParams {
  to: string;
  from: string;
}

const { sendEmail } = proxyActivities<ReturnType<typeof createActivities>>({
  retry: {
    maximumAttempts: 2
  },
  startToCloseTimeout: '5m',
});

export async function customerWelcomeSeriesWorkflow(params: CustomerWelcomeSeriesParams): Promise<void> {
  const { from, to } = params;

  // Assume app sends a signal to this Workflow when a customer logs in
  let hasLoggedIn = false;
  setHandler(loggedInSignal, () => {
    hasLoggedIn = true;
  });

  await sendEmail({
    to,
    from,
    subject: 'Welcome to Temporal!',
    html: 'This is a test email'
  });

  await sleep('1 days');

  // Send a different email if user hasn't logged in during first 24 hours
  if (!hasLoggedIn) {
    await sendEmail({
      to,
      from,
      subject: 'Looks Like You Haven\'t Tried Temporal Yet...',
      html: 'This is a test email'
    });
  } else {
    await sendEmail({
      to,
      from,
      subject: 'Your First Day Using Temporal',
      html: 'This is a test email'
    });
  }
}
```

While Temporal Workflows can execute just one task, they can also execute a whole series of tasks. With a conventional task queue, you would have to schedule a separate task for each email in `customerWelcomeSeriesWorkflow()`. Temporal Workflows make writing multi-part tasks much easier, because you can write the task orchestration logic in a single JavaScript function, rather than spread out across multiple task handlers.

## Moving On

Temporal is not a task queue, but you can use Temporal Workflows to solve the same problem that task queues solve: sending tasks to workers using a queue in a durable and observable way. Temporal provides a lot of the same features as task queues, including retries, delays, and a sleek UI. But Temporal Workflows also help you write more complex multi-step tasks in a single JavaScript function, including logic that would require multiple tasks in a traditional task queue. So next time you're looking for a Node.js task queue, try Temporal instead!


node-js-task-queue

While Temporal is much more than a task queue, Temporal does solve the same problems as these task queue libraries.

Using Temporal as a Node.js Task Queue

Community Member

Valeri Karpov


A [Temporal Workflow Execution](https://docs.temporal.io/workflows#workflow-executions) has no time limit. You can write a Workflow that runs forever, storing some state and responding to Signals and Queries, as long as you remember to use [Continue-As-New](https://docs.temporal.io/workflows#continue-as-new). One neat use case for long-lived Workflows is caching API requests.

For example, suppose you want to display prices in various currencies based on cached exchange rates. Exchange rate APIs are often expensive for high volumes of requests, so caching can be worthwhile as long as stale data isn't a problem for your use case. You can create a single Temporal Workflow that makes one API request per day to get the latest exchange rates for a set of currencies and stores the most recent result, with no explicit database calls or cron jobs. In this blog post, I'll describe how you can write an API caching Workflow for the [moneyconvert.net API](https://moneyconvert.net/) with the Temporal TypeScript SDK. You can find the full source code for this example in [vkarpov/temporal-api-caching-example](https://github.com/vkarpov15/temporal-api-caching-example) on GitHub.

## Getting Started

When you make an HTTP GET request to https://cdn.moneyconvert.net/api/latest.json, the API returns JSON that looks like the following:

```ts
{
  "table": "latest",
  "rates": {
    "AED": 3.6731,
    "AFN": 87.249991,
    "ALL": 114.227196,
    "AMD": 473.915796,
    ...
  },
  "lastupdate": "2022-05-11T00:46:04.483000+00:00"
}
```

Because Temporal Workflows must be deterministic, you can't make an API request directly from a Workflow. You need to create an Activity that makes an API request, and call that Activity from your Workflow. Following is an [`activities.ts`](https://github.com/vkarpov15/temporal-api-caching-example/blob/main/src/activities.ts) file that makes an HTTP request to the preceding API endpoint and returns the exchange rates.

```ts
import axios from 'axios';

export async function getExchangeRates(): Promise<any> {
  const res = await axios.get('https://cdn.moneyconvert.net/api/latest.json');
  return res.data.rates;
};
```

Next, let's write a Workflow that calls this Activity once per day. With Temporal Workflows, you can simply write a `while (true) {}` loop with a [JavaScript sleep](https://masteringjs.io/tutorials/fundamentals/sleep) that pauses the Workflow until the next time you need to refresh exchange rates. Writing the Workflow this way might seem counterintuitive, because we've all had to learn over the course of our careers that writing applications isn't this easy. But, with Temporal, it actually is this easy!

```ts
import {
  defineQuery,
  proxyActivities,
  setHandler,
  sleep
} from '@temporalio/workflow';
import type * as activities from './activities';

const { getExchangeRates } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

type ExchangeRatesType = { [key: string]: number };

export const getExchangeRatesQuery = defineQuery<ExchangeRatesType | undefined, [string]>('getExchangeRates');

export async function exchangeRatesWorkflow(): Promise<void> {
  let rates: ExchangeRatesType | undefined = undefined;

  // Register a query handler that allows querying for the current rates
  setHandler(getExchangeRatesQuery, () => rates);

  while (true) {
    // Get the latest rates
    rates = await getExchangeRates();

    // Sleep until tomorrow at 12pm server time, and then get the rates again
    const today = new Date();
    const tomorrow = new Date(today);
    tomorrow.setHours(12, 0, 0, 0);
    tomorrow.setDate(tomorrow.getDate() + 1);
    await sleep(tomorrow.valueOf() - today.valueOf());
  }
}
```

That's the entire Workflow for caching exchange rates. The full source code is available in [this GitHub repo](https://github.com/vkarpov15/temporal-api-caching-example). Notice that the code has no explicit references to a database or job queue. This Workflow is almost pure business logic, with a minimum of references to frameworks or services.

## Workflow Queries

To run this Workflow, you can run a `start-workflow.ts` script as shown below. This script starts a Workflow and exits, leaving the Workflow running on the Worker. Note that only one Workflow with a given `workflowId` can run at any given time, so the following code also ensures that only one copy of this Workflow is running at any particular time.

```ts
import { WorkflowClient } from '@temporalio/client';
import { exchangeRatesWorkflow } from '../workflows';

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

async function run() {
  const client = new WorkflowClient();

  const handle = await client.start(exchangeRatesWorkflow, {
    taskQueue: 'exchange-rates',
    workflowId: 'exchange-rates-workflow',
  });
  console.log(`Started workflow ${handle.workflowId}`);
}
```

Following is the output of `start-workflow.ts`.

```
Started workflow exchange-rates-workflow
```

After you start the Workflow, you can execute a Query to get the latest exchange rates using the following `query-workflow.ts` script.

```ts
import { WorkflowClient } from '@temporalio/client';
import { getExchangeRatesQuery } from '../workflows';
import { toYYYYMMDD } from '../shared';

const date: string = process.argv[2] || toYYYYMMDD(new Date()); 

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

async function run() {
  const client = new WorkflowClient();

  const handle = client.getHandle('exchange-rates-workflow');
  console.log('Query exchange rates for', date);
  console.log(await handle.query(getExchangeRatesQuery, date));
}
```

Following is an example of the output from running `query-workflow.ts`.

```
Query exchange rates for 20220527
{
 AED: 3.6731,
 AFN: 88.999995,
 ALL: 112.95,
 AMD: 455.70744,
 ...
}
```

This Workflow is missing one key detail: a [Continue-As-New](https://docs.temporal.io/workflows#continue-as-new). There's more about that later in this blog post.

## Storing Historical Data

You can do more than just store the latest exchange rates. You can also store previous exchange rates. For example, suppose you want to store up to 30 days worth of historical exchange rates. You can store the rates in an in-memory JavaScript map in your Workflow, as shown in the following code.

```ts
const maxNumRates = 30;

export async function exchangeRatesWorkflow(): Promise<void> {
  const ratesByDay = new Map<string, ExchangeRatesType>();

  // Allow querying exchange rates by day
  setHandler(getExchangeRatesQuery, (date: string) => {
    return ratesByDay.get(date);
  });

  while (true) {
    const exchangeRates = await getExchangeRates();
    const today = new Date();
    // Store today's exchange rates
    ratesByDay.set(toYYYYMMDD(today), exchangeRates);
    console.log(toYYYYMMDD(today), exchangeRates);

    // Delete the oldest key if we have more than 30 entries
    const keys = Array.from(ratesByDay.keys());
    if (keys.length > maxNumRates) {
      ratesByDay.delete(keys[0]);
    }

    // Wait until tomorrow at 12pm to refresh the exchange rates
    const tomorrow = new Date(today);
    tomorrow.setHours(12, 0, 0, 0);
    tomorrow.setDate(tomorrow.getDate() + 1);
    await sleep(tomorrow.valueOf() - today.valueOf());
  }
}
```

Temporal makes `ratesByDay` durable, even though `ratesByDay` is just a normal JavaScript variable. That's because Temporal stores the entire history of events for this Workflow. If the machine running `exchangeRatesWorkflow()` crashes, Temporal can resume the Workflow on another machine by replaying the entire event history.

## Continue-As-New

The `exchangeRatesWorkflow` can run for an unlimited period of time: days, months, even years. However, Temporal caps a Workflow at 50,000 events. (See the [Time constraints](https://docs.temporal.io/workflows#time-constraints) section in [Temporal Workflows](https://docs.temporal.io/workflows).) In the `exchangeRatesWorkflow`, eight events are fired during each iteration of the `while` loop, assuming no API errors.

1. `EVENT_TYPE_TIMER_FIRED`: the `setTimeout()` resolved and it's time to refresh the exchange rates
2. `EVENT_TYPE_ACTIVITY_TASK_SCHEDULED`: the Temporal server scheduled the `getExchangeRates()` activity
3. `EVENT_TYPE_ACTIVITY_TASK_STARTED`: the `getExchangeRates()` activity started executing
4. `EVENT_TYPE_ACTIVITY_TASK_COMPLETED`: the `getExchangeRates()` activity completed successfully
5. `EVENT_TYPE_WORKFLOW_TASK_SCHEDULED`: the Temporal server scheduled the Workflow logic that handles `getExchangeRates()`
6. `EVENT_TYPE_WORKFLOW_TASK_STARTED`: the Workflow resumed
7. `EVENT_TYPE_WORKFLOW_TASK_COMPLETED`: the Workflow paused
8. `EVENT_TYPE_TIMER_STARTED`: the Workflow used `setTimeout()` to pause until tomorrow

With one API request per day, `exchangeRatesWorkflow()` can run for almost 6,250 days (approximately 17 years) before running into the 50,000 event limit. However, you should still avoid running into this limit. And that's what Continue-As-New is for.

You can think of Continue-As-New as restarting your Workflow from an initial state. The only data that `exchangeRatesWorkflow()` needs to respond to queries is the `ratesByDay` map, so `exchangeRatesWorkflow()` needs to Continue-As-New with a serialized version of the `ratesByDay` map. The `exchangeRatesWorkflow()` also needs to be able to resume from a previous state. Continue-As-New just calls `exchangeRatesWorkflow()` with an initial state. Following is the `exchangeRatesWorkflow()` workflow with an extra `storedRatesByDay` parameter that will contain the serialized Workflow state after a Continue-As-New.

```ts
const maxNumRates = 30;
const maxIterations = 1000;

// `storedRatesByDay` contains the serialized data from Continue-As-New, if available. Otherwise, just an
// empty array.
export async function exchangeRatesWorkflow(storedRatesByDay: Array<[string, any]> = []): Promise<any> {
  const ratesByDay = new Map<string, any>(storedRatesByDay);
  setHandler(getExchangeRatesQuery, (date: string) => {
    return ratesByDay.get(date)
  });

  // Max out at 1k iterations (~8k events) so we don't get too close to the 50k event limit
  for (let i = 0; i < maxIterations; ++i) {
    const exchangeRates = await getExchangeRates();
    const today = new Date();
    ratesByDay.set(toYYYYMMDD(today), exchangeRates);

    tomorrow.setHours(12, 0, 0, 0);
    tomorrow.setDate(tomorrow.getDate() + 1);
    await sleep(tomorrow.valueOf() - today.valueOf());
  }

  // After 1k iterations, trigger a Continue-As-New and finish the Workflow
  const state = Array.from(ratesByDay.entries());
  await continueAsNew<typeof exchangeRatesWorkflow>(state);
}
```

## Moving On

Temporal Workflows make it easy to build stateful caching layers for APIs. With Temporal, you just write the business logic. Temporal handles persisting the data, ensuring only one copy of the Workflow is running, and allowing clients to query the cached API data. Next time you need to build a tool that pings an API on a regular schedule, try building it with Temporal.


caching-long-running-workflows

You can write a Workflow that runs forever, storing some state and responding to Signals and Queries, as long as you remember to use Continue-As-New. One neat use case for long-lived Workflows is caching API requests.

Caching API Requests With Long-Lived Workflows in Temporal

I previously wrote about using Temporal's long-lived Workflows to build [REST APIs without a conventional database](https://docs.temporal.io/blog/build-an-ecommerce-app-with-temporal-part-4-rest-api), but the REST layer was written manually and with a lot of boilerplate.

While you can always build a REST API on top of Temporal Workflows yourself, I've now built [`temporal-rest`](https://www.npmjs.com/package/temporal-rest) to make creating a RESTful API for your Workflows a one-liner with [ExpressJS](https://expressjs.com/).
Just add middleware!

In this blog post, I'll show how you can use `temporal-rest` to easily create RESTful APIs backed by long-lived Workflows.

## Temporal without REST

The key idea of long-lived Workflows is that Workflow functions are _deterministic_, so Temporal can persist the state of the Workflow by storing the Workflow's initial state and event history. For example, below is a Workflow that keeps a single numeric counter. The Workflow listens for an `increment` Signal to increase the counter, and a `getCount` Query to get the current value of the counter.

```ts
import * as wf from '@temporalio/workflow';

export const incrementSignal = wf.defineSignal('increment');
export const getCountQuery = wf.defineQuery('getCount');

export async function counterWorkflow(): Promise<void> {
  let count = 0;

  wf.setHandler(exports.incrementSignal, () => ++count);
  wf.setHandler(exports.getCountQuery, () => count);

  // Wait forever
  await wf.condition(() => false);
}
```

To execute an instance of `counterWorkflow`, use `WorkflowClient` to start the Workflow:

```ts
import { WorkflowClient } from '@temporalio/client';
import { counterWorkflow } from './workflows';
import crypto from 'crypto';

async function run() {
  const client = new WorkflowClient();

  const handle = await client.start(counterWorkflow, {
    taskQueue: 'tutorial',
    // Usually, we use a business ID. In this example, we don't have one, so we generate an ID.
    workflowId: 'counter-' + crypto.randomUUID(),
  });

  // Increment the counter
  await handle.signal('increment');

  // Prints "1"
  console.log(await handle.query('getCount'));
}

run().catch(err => {
  console.error(err);
  process.exit(1);
});
```

## Introducing `temporal-rest`

Running `counterWorkflow` from this command-line script is convenient for the sake of an example, but not very useful unless you're building a command-line app. Enter `temporal-rest`, which you can use to create an Express API for `counterWorkflow`:

```ts
import { WorkflowClient } from '@temporalio/client';
import * as workflows from './workflows';

import express from 'express';
import { createExpressMiddleware } from 'temporal-rest';

async function run() {
  const client = new WorkflowClient();

  const app = express();

  // this is the oneliner that does all the work!
  app.use(createExpressMiddleware(workflows, client, 'tutorial')); 

  await app.listen(3000);
  console.log('Listening on port 3000');
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

The `createExpressMiddleware()` function creates an Express router with 3 endpoints:

1. `POST /workflow/counterWorkflow`: start a new instance of `counterWorkflow`.
2. `GET /query/getCount/:workflowId`: execute the `getCount` Query on the Workflow with the given ID.
3. `PUT /signal/increment/:workflowId`: send an `increment` Signal to the Workflow with the given ID.

Below is an example of interacting with this API using [`curl`](https://thecodebarbarian.com/what-javascript-developers-should-know-about-curl.html). A POST to `/workflow/counterWorkflow` creates a new Workflow instance and returns the Workflow ID. Then you can send a Signal to increment the counter, and execute a Query to get the current state of the counter:

```ts
$ curl -X POST http://localhost:3000/workflow/counterWorkflow
{"workflowId":"4cb1b1ea-b962-419e-840c-5c18ab5555a1"}$ 
$ curl "http://localhost:3000/query/getCount/4cb1b1ea-b962-419e-840c-5c18ab5555a1"
{"result":0}
$ curl -X PUT "http://localhost:3000/signal/increment/4cb1b1ea-b962-419e-840c-5c18ab5555a1"
{"received":true}
$ curl "http://localhost:3000/query/getCount/4cb1b1ea-b962-419e-840c-5c18ab5555a1"
{"result":1}
```

## REST endpoints for Queries and Signals

You can also invoke Signals and Queries with `temporal-rest`. By default, `temporal-rest` parses any JSON in the [Express request body](https://masteringjs.io/tutorials/express/body) and passes the parsed object as the first parameter to Signals, and passes the [Express query parameters](https://masteringjs.io/tutorials/express/query-parameters) as the first parameter to Queries.

For example, suppose `counterWorkflow` supports tracking multiple counters, each one with a unique name:

```ts
import * as wf from '@temporalio/workflow';

export const incrementSignal = wf.defineSignal('increment');
export const getCountQuery = wf.defineQuery('getCount');

export async function counterWorkflow(): Promise<void> {
  const counters = new Map<string, number>();

  wf.setHandler(incrementSignal, (args: { name: string }) => {
    const count = counters.get(args.name);
    if (count !== undefined) {
      counters.set(args.name, count + 1);
    } else {
      counters.set(args.name, 1);
    }
  });
  wf.setHandler(getCountQuery, (args: { name: string }) => {
    if (!counters.has(args.name)) {
      return 0;
    }
    return counters.get(args.name);
  });

  // Wait forever
  await wf.condition(() => false);
}
```

To create a new counter with this Workflow, we would normally send an `increment` Signal with an object containing the counter's `name`:

```ts
const handle = await client.start(counterWorkflow, {
  taskQueue: 'tutorial',
  workflowId: 'counter-' + crypto.randomUUID(),
});

// Increment a new counter
await handle.signal('increment', { name: 'test-counter' });

console.log(await handle.query('getCount', { name: 'test-counter' }));
// => "1"

console.log(await handle.query('getCount', { name: 'other-counter' }));
// => "0" because there's no counter named 'other-counter'
```

Instead, we can now expose these Signals via a RESTful API with the `temporal-rest` middleware the same as we did above:

```ts
  app.use(createExpressMiddleware(workflows, client, 'tutorial'));
```

To pass a parameter to the `increment` Signal, we pass a JSON-encoded HTTP request body to the Signal's POST endpoint, and to pass a parameter to the `getCount` Query, we add a query string to the Query's GET endpoint.
Here's an example using `curl`:

```ts
$ curl -X POST http://localhost:3000/workflow/counterWorkflow
{"workflowId":"cbc5924c-1afc-45e0-b7d6-e8fe1a250089"}
$ curl -X PUT -H "Content-Type: application/json" -d '{"name":"test-counter"}' http://localhost:3000/signal/increment/cbc5924c-1afc-45e0-b7d6-e8fe1a250089
{"received":true}
$ curl http://localhost:3000/query/getCount/cbc5924c-1afc-45e0-b7d6-e8fe1a250089?name=test-counter
{"result":1}
$ curl http://localhost:3000/query/getCount/cbc5924c-1afc-45e0-b7d6-e8fe1a250089?name=other-counter
{"result":0}
```

Here's an alternative example of making requests to this API using the [Axios HTTP client](https://masteringjs.io/axios) in Node.js:

```ts
let res = await axios.post('http://localhost:3000/workflow/counterWorkflow');
console.log(res.data); // "{ workflowId: '5232d34e-1c65-4d38-8470-39ec03b0eb02' }"
const { workflowId } = res.data;

res = await axios.put('http://localhost:3000/signal/increment/' + workflowId, {
  name: 'test-counter'
});
console.log(res.data); // "{ received: true }"

res = await axios.get('http://localhost:3000/query/getCount/' + workflowId, {
  params: { name: 'test-counter' }
});
console.log(res.data); // "{ result: 1 }"
```

## Moving On

Long-lived Workflows in Temporal let you build durable, scalable RESTful APIs without a traditional database. The [`temporal-rest`](https://www.npmjs.com/package/temporal-rest) package removes the boilerplate of wrapping Temporal Workflows, Queries, and Signals in an API. Just write your Workflows, and `temporal-rest` takes care of the Express API. Try `temporal-rest` out and let us know what you think in the comments or [GitHub issues](https://github.com/vkarpov15/temporal-rest/issues).


While you can always build a REST API on top of Temporal Workflows yourself, I've now built temporal-rest to make creating a RESTful API for your Workflows a one-liner with ExpressJS. 

REST APIs for every Temporal Workflow in one line of code

For the past 45 years, the database community has enjoyed an unparalleled developer experience: transactions mitigate failure in totality on a platform level, guaranteeing correctness on an application level.

Despite many advancements in the past 20 years, the distributed systems community has not enjoyed an equivalent developer experience: There is no abstraction that mitigates failure in totality on a platform level, guaranteeing correctness on an application level.

However, [Temporal](https://temporal.io/) changes that equation! 

## Introduction

Temporal's core abstraction, its unit of execution, reliability, and scalability, is the Workflow. Therefore understanding the Workflow is key to understanding Temporal in general. In this blog post series we deep dive into the world of Temporal Workflows.

A Temporal Workflow is Temporal's core abstraction. You may think of a Temporal Workflow Definition as a regular Function Definition—in fact, that is the developer experience that Temporal provides to its users—but a Workflow Execution provides stunning improvements over a regular Function Execution.

Let's get to know Workflow Executions and contrast them to regular Function Executions with a straightforward example: sending reminder emails.

Our use case requires that our application sends a reminder email once a month to any user who signed up for a trial period to upgrade their plan. In pseudo code our use case can be expressed as:

```
function send monthly reminder (user) do
  while user has not signed up do
    send reminder to user
    sleep for 1 month
  end
end
```

## Regular Functions

The pseudo code looks fairly straightforward; in fact, the pseudo code looks less like an implementation and more like a specification. Could we use the pseudo code as a blueprint for a regular Function Definition?

No, not at all:

> In a typical environment we cannot just invoke a function and expect the resulting function execution to reliably execute to completion — or like in this case execute indefinitely until cancelation.

As a result, we have to "break up" the process of Send Reminder Email into many different pieces, scattered across the tech stack: A cron job here, a message in a queue there, maybe a row in some database table, you know, for good measure.

On top of that, now we need to worry about failures, retries, duplication, and idempotence.

An implementation on top of services like AWS Lambda Functions and like AWS Simple Queueing Service might look like:

```typescript

// Lambda function is bound to
// a. input queue "Reminder"
// b. output queue "Reminder"

// We assume that
// - messages are never lost
// - messages may be duplicated
// - messages are retried on failure

function SendReminderEmail(event, context) : Message {

  // UserSignup, SendEmail, Get, Set will throw an
  // Exception on failure

  // event.user Current user
  // event.iter Current iteration to limit retries (here 2)

  if (!UserSignedUp(event.user)) {

    // Retrieve the k/v pair for this user and iteration
    let kv = Get(`$(event.user)-${event.iter}`, 0);

    // Try at most twice
    if(kv.val < 2) {
      // Conditionally set the key. If the tag does not 
      // match we are racing with another instance of
      // SendReminderEmail
      if (Set(kv.key, kv.val + 1, kv.tag)) {
        // This does not prevent us from calling SendEmail
        // twice. Do you see why?
        SendEmail(user);
      }
    }
    else {
      throw;
    }
  }

  return {
    message: { user: event.user, iter: event.iter + 1},
    after: "30 days"
  }
}

// Start by queueing the message {user: "<User>", iter: 0} on
// the message queue "Reminder"

```

Listing 2 looks nothing like the pseudo code in Listing 1. 

Listing 2 does not tell the story of our use case—while not overly long or verbose, it is obscure and hard to reason about.

## Temporal Workflows

Obviously we cannot use the pseudo code as a blueprint for a regular Function Definition. However, could we use the pseudo code as a blueprint for a Temporal Workflow Definition?

Well, yes, yes we can!

```typescript
import { proxyActivities, sleep } from '@temporalio/workflow';

const { sendReminderEmail, hasSignedUp } = proxyActivities({
  scheduleToCloseTimeout: '10 seconds',
  retry: { maximumAttempts: 2 }
});

async function SendReminderEmail(user: string) {
    while(!await hasSignedUp(user)) {
      try {
        await sendReminderEmail(user);
      } catch(e) {
        // Thanks to Temporal's retry policy, we already 
        // tried twice, better luck next month 🍀
      }
      await sleep("30 days");
    }
}
```

> In Temporal we can just invoke a Workflow Definition, and the resulting Workflow Execution reliably executes to completion — or like in this case execute indefinitely until cancelation.

Temporal Workflow Executions are to distributed systems what transactions are to databases: A great developer experience and (or maybe because of) peace of mind.

Doubts? Disbelief? Check out Part II and Part III to explore how Temporal implements this game-changing execution model.

Photo by [Simon Berger](https://unsplash.com/@8moments?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/zen?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)


hhvrdmolw4m39v0vqvrh

For the past 45 years, the database community has enjoyed an unparalleled developer experience: transactions mitigate failure in totality on a platform level, guaranteeing correctness on an application level.

Introduction to Temporal Workflows

Principal Engineer

Dominik Tornow

dominik-tornow

> Update: See also our new docs on [SDK metrics](https://docs.temporal.io/references/sdk-metrics/) and [Temporal Workers Tuning](https://docs.temporal.io/application-development/worker-performance/) for detailed instructions.

Whether you are self hosting Temporal Server or using Temporal Cloud, a key part of the operational model of Temporal is running your own Workers. We recently hosted a chat with Temporal CTO Samar Abbas on the 4 main areas to note when productionizing your Temporal Workers:

- Operational Metrics - Monitoring and Alerting
- Incident Response & Tooling
- Upgrading and Versioning
- Testing Failure Paths

[Watch the full presentation here.](https://www.youtube.com/watch?v=bKRIkbxrVjs)

## Operational Health Metrics - Monitoring and Alerting (02:30)

The great thing about Temporal is that you can develop against it locally without constraints (and it is getting even faster with [temporalite](https://github.com/DataDog/temporalite)). But when it comes to running your Temporal applicaiton in production, you are now dealing with a distributed system.

**In other words: How will you know if your Temporal application is doing its job or is even running as expected?**

Temporal emits a lot of metrics, both client and server, which gives insights into how your application is performing: counters, latency, and metrics for server health and for each namespace. Temporal uses the [Tally library from Uber](https://github.com/uber-go/tally) to emit properly buffered metrics with nice reporting integration with m3, Prometheus, Statsd, and [Datadog via custom reporters](https://docs.temporal.io/server/options/#withcustommetricsreporter), and provides [dashboards for Grafana users](https://github.com/temporalio/dashboards) to display them. We also recently co-presented about [Temporal and M3 with Chronosphere at KubeCon NA 2021](https://www.youtube.com/watch?v=8OCDPDGA_C0).

- **Activity Worker Health**
    - How many activities are getting started?
    - How many activities are completing successfully?
    - Which activities are failing?
    - **Set alerts for Activity failure rate**
- **Workflow Worker Health**
    - What rate are your Workflow Tasks happening at?
    - Are your Workflow Tasks completing successfully?
    - What is your Workflow Task latency?
    - **Set alerts for Workflow Task failure rate**
- **Scaling**
    - **Set alerts on [Activity Schedule to Start latency and Workflow Schedule to Start latency](https://docs.temporal.io/server/production-deployment/#scaling-and-metrics)** and scale your Workers based on that

Our full reference on [SDK/Worker metrics is here](https://docs.temporal.io/references/sdk-metrics/).

When you have ingested all these metrics and have set up a dashboard to give you visibility, at least now you can start getting a n overall picture of what’s going on with the system.

## Incident Reponse & Tooling (10:06)

When Workflow Task Failure rates spike, you are now in incident response mode. There can be many possible causes, so the first thing an operator needs to do is to figure out whether the failure is from your application or Temporal Server. 

- **Metrics**. The first metric to look at is the “service request” and “service error” metrics emitted by the Server. If you are seeing workflow tasks or activities failing, it is a strong indication that you should focus on the application side rather than Temporal Server.
- **Logs**. The best tool for investigating application side failures is your logs - make sure your logs are stored and captured by your workers, and tagged with important structured data like workflow type, activity type, workflow ID, and run ID. Try to put as much information as tags to allow you to slice and dice logs on arbitrary criteria.
- **Execution History**. You should be able to tell whether your failures are coming from one particular workflow type, and to be able to isolate a specific workflow/run ID to investigate. You can then use **Temporal Web or Temporal CLI (tctl)** to look at the state and execution history of that workflow, which will reveal more datapoints on how to debug. In the history view, you should be able to see the entire call stack of the failure as well.
- **Replay**. You can also **replay the Workflow** to investigate it, by downloading the execution history via the Web or CLI tool, and stepping through the code using a debugger on your local machine.

## Upgrading and Versioning (17:59)

How do you migrate long running workflows that are still in flight? Most developers who hand roll their own job scheduling and orchestration frameworks don’t have anything better than “deploy and pray”, but Temporal offers a first class solution.

Temporal requires that your Workflows must be fully deterministic, and changing Workflow code means potentially breaking determinism. This is where the `getVersion` API is handy because it allows you to make changes in Workflow logic in a deterministic fashion.

Temporal offers a `getVersion` API in the [Go](https://docs.temporal.io/go/versioning/#workflowgetversion), [Java](https://docs.temporal.io/java/versioning/#java-versioning-api), [PHP](https://docs.temporal.io/php/versioning/#workflowgetversion) SDKs, and a [Patching](https://docs.temporal.io/typescript/patching) API in TypeScript, that essentially amounts to feature flagging by start time cohort:

```go
v := workflow.GetVersion(ctx, "Step1", 1, 2)
if v == 1 {
        err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1)
} else {
        err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1)
}
```

We have also published a **[30 min tutorial on versioning](https://www.youtube.com/watch?v=kkP899WxgzY&feature=emb_title)** to guide you through this API.

- **How it works**: Temporal internally tracks the version of Workflow code at the time that particular Workflow Execution was started, and uses that to navigate code paths based on these user defined branches. New Workflow Executions take the latest paths by default.
- **Sample and Replay:** While this is an elegant and powerful API (because it can be used and composed in arbitrary logic), it still needs to be handled carefully. This is why replaying workflow execution is so important. Before migration, take samples of Workflow histories at different stages, and use the replay API.
- **Use Task Queues for Short-Lived Workflows**: Think through your upgrade strategy - you may not even need versioning if your workflows are short-lived enough! If you just want to cut over to new code, you can run old code on the same Task Queue with a subset of Workers, and run new code on a different set of Workers with a new Task Queue!
- **Worst Case**: When in doubt - take a breath - because of Temporal’s event sourced nature, if a migration is screwed up, **our worst case is that your Workflows don’t make progress** - latency will spike, but data will not be lost - just make sure you are fully familiar with metrics and logs to investigate issues!

## Testing Failure Paths (24:38)

Most people (including us) primarily code for happy paths when thinking about business logic. However there are some unhappy paths that are persistently overlooked:

- **Backlogs**: A backlog happens when multiple Workflow tasks (mainly Timers) resolve together when some part of your system (say your Workers, or even the Server) is down. When Temporal recovers, it will still fire those timers, but it is not guaranteed to fire them in chronological order. However, happy-path code will typically assume that they do. **Always** test your system for backlogs.
    - Specifically, when you write end-to-end tests for your production Temporal system, make sure to write some that **shuts down your Workers** for some time to create backlog of Workflow tasks, and then spin them up again to churn through the backlog. You can do it [programmatically within an SDK Worker API](https://docs.temporal.io/typescript/workers#how-to-shut-down-a-worker-and-track-its-state) or externally through your infra automation tool of choice.
- **Capacity Planning**: End to end tests that include Worker outages not only help find edge cases like backlogs. When the system has a backlog, the system’s resource consumption is very different than under normal circumstances. Always do capacity planning accounting for cases where the system is going through heavy load due to backlog or spikes.
    - Note: we have also written about [how we do stress testing](https://docs.temporal.io/blog/temporal-deep-dive-stress-testing/) and [Maru is a third party load testing tool](https://mikhail.io/2021/03/maru-load-testing-tool-for-temporal-workflows/).
- **Timeouts**: Temporal provides [4 kinds of Activity timeouts](https://docs.temporal.io/blog/activity-timeouts/) (we don’t recommend using Workflow timeouts) and a declarative [Retry Policy](https://docs.temporal.io/concepts/activities/#retries) which are very flexible, which is powerful but can be a source of mistakes. We recommend explicitly unit- and integration-testing for these timeouts - don’t assume that the code you have written will behave as you intended.

Did this discussion help with your production concerns? What other questions to you wish to ask? Email [community@temporal.io](mailto:community@temporal.io) to request the next topic!


nkululeko-jonas-fSeKcdzeZTE-unsplash

Whether you are self hosting Temporal Server or using Temporal Cloud, a key part of the operational model of Temporal is running your own Workers.

Productionizing Workers in Temporal

Developer Relations

swyx

We often say that "Temporal Server is a simple Go binary", and [it's true](https://github.com/temporalio/temporal/releases/): Each release includes 4 docker images, but also zipped versions of our `temporal-server` binary.

![148215905-ffb15e1a-75ba-4ddc-aa91-12d9e374f2d1](//images.ctfassets.net/0uuz8ydxyd9p/1tzR7RPMNtFsbZSMjGoJzz/e9a7da354eae719f950218eb3727a4f6/148215905-ffb15e1a-75ba-4ddc-aa91-12d9e374f2d1.png)

Yet most users are insulated from using this binary directly. There are several layers of abstraction to get through ([docker-compose](https://github.com/temporalio/docker-compose) or [Helm Chart](https://github.com/temporalio/helm-charts), Dockerfile, and the `auto-setup.sh` script) before you reach this baseline understanding of Temporal (you can [learn more in our Temporal Explained documentation](/temporal). Together they map out the environment requirements of Temporal:

![147678999-883be1b4-4d32-4c89-84a4-00e8b701cdef](//images.ctfassets.net/0uuz8ydxyd9p/2U0hnmDuqDywrtYBbbwgNY/7b217e0e2ec1daa9bc54c82c75d3ee7b/147678999-883be1b4-4d32-4c89-84a4-00e8b701cdef.png)

While the other pieces are industry standard formats with explanations available elsewhere, the `auto-setup` script is the most proprietary of them, and so essential that we often recommend beginners just use the [temporalio/auto-setup](https://hub.docker.com/r/temporalio/auto-setup) Docker image for getting started (most beginners don't even know this because we abstract it with docker-compose), before migrating to just the Temporal Server Docker image without auto-setup for production (where you can account for schema migrations and other tooling needs specific to your environment).

It's worth:

- knowing that the `auto-setup.sh` script exists and handles some critical pieces of the Temporal deployment model
- understanding what it does to fill in the blanks between "it's just a binary" to a zero configuration [Quick Install](https://docs.temporal.io/clusters/quick-install) default developer experience
- understanding what is optional - so you can modify it when it is getting in your way, or have confidence throwing it out and writing your own

The goal of this post is ***not*** to explain every little detail (you can [read the code](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) yourself for that), but to highlight important pieces every self-hosted Temporal user should know.
This should give you some great first-principles knowledge to deploy Temporal to anything from a simple VPS to a full Kubernetes cluster.

## Auto-Setup at a High Level

Our [basic Docker entrypoint](https://github.com/temporalio/docker-builds/blob/main/docker/entrypoint.sh) does some bindings, but most importantly, it runs [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) then [start-temporal.sh](https://github.com/temporalio/docker-builds/blob/main/docker/start-temporal.sh).  

`auto-setup.sh` does a few things (we will go deeper on the **bolded** parts):

1. Temporal Schema Setup (if you don't opt out):
    1. Validates seed environment variables (trivial step)
    2. Waits for the database to start (using `nc` for SQL databases, or [`temporal-cassandra-tool`](https://github.com/temporalio/temporal/blob/master/tools/cassandra/README.md) for Cassandra)
    3. **Setup Temporal Schema on that database (more on this below)**
2. Elasticsearch Schema Setup (if you opt in):
    1. Validates seed environment variables (trivial step)
    2. Waits for Elasticsearch to start (optionally, timed out with a `ES_SCHEMA_SETUP_TIMEOUT_IN_SECONDS` env var)
    3. Setup Temporal's Enhanced Visibility Schema on that Elasticsearch instance
3. Setup Temporal Server:
    1. Wait for Temporal Server to start (from running [start-temporal.sh](http://start-temporal.sh) concurrently, and checked using `tctl cluster health | grep SERVING`
    2. **Register the `default` namespace** (unless opted out with `SKIP_DEFAULT_NAMESPACE_CREATION`)
    3. Add basic custom search attributes (unless opted out with `SKIP_ADD_CUSTOM_SEARCH_ATTRIBUTES`)

<details>
<summary>
Note that this script only handles setup - it doesn't actually run the `temporal-server` binary itself. That job is given to `start-temporal.sh`, whose main job is to run `temporal-server` with flags for given service roles (mainly specified by the `SERVICES` env var).
</summary>

Note that when we say to run each of these services separately in production so they can be independently scaled, this is the exact step where production differs from local development. You can run the same binary in multiple separate processes with different roles for each, as in:

```tsx
# process 1
temporal-server start --service=history

# process 2
temporal-server start --service=frontend

# process 3
temporal-server start --service=matching

# process 4
temporal-server start --service=worker
```

You may also use the Docker container with the appropriate flags:

```bash
docker run -e CASSANDRA_SEEDS=10.x.x.x                  -- csv of cassandra server ipaddrs
    -e KEYSPACE=<keyspace>                              -- Cassandra keyspace
    -e VISIBILITY_KEYSPACE=<visibility_keyspace>        -- Cassandra visibility keyspace
    -e SKIP_SCHEMA_SETUP=true                           -- do not setup cassandra schema during startup
    -e NUM_HISTORY_SHARDS=1024  \                       -- Number of history shards
    -e SERVICES=history,matching \                      -- Spinup only the provided services
    -e LOG_LEVEL=debug,info \                           -- Logging level
    -e DYNAMIC_CONFIG_FILE_PATH=config/foo.yaml         -- Dynamic config file to be watched
    temporalio/server:<tag>
```

</details>

> Side note: It is called "auto setup" because [Márk Sági-Kazár](https://www.linkedin.com/feed/update/urn:li:activity:6885550647865958400?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6885550647865958400%2C6891543183516667904%29), one of our early external contributors, coined it!

## Temporal Schema Setup

Database schema setup is only needed once for new databases being set up to work with Temporal Server, and is done with [temporal-sql-tool](https://github.com/temporalio/temporal/blob/f355adc9505123c63ed1ad888449dcb89584f8cd/tools/sql/README.md) or [temporal-cassandra-tool](https://github.com/temporalio/temporal/tree/f355adc9505123c63ed1ad888449dcb89584f8cd/tools/cassandra) depending on the type of database specified.

- Together with the `tctl` CLI, these binaries are available in the [temporalio/admin-tools](https://hub.docker.com/r/temporalio/admin-tools) Docker image.
- These tools are also used for updating schemas when new Temporal versions are being rolled out to existing deployments.

The schema setup process is pretty similar across all database types:

1. Create a new Temporal database with `DBNAME`
2. Run the `setup-schema` command with the `temporal-[db]-tool`
3. Run the `update-schema` command with the `temporal-[db]-tool` with a specific schema for that combination of database + schema version  
4. Create a new Standard Visibility database with `VISIBILITY_DBNAME`
5. Run the `setup-schema` command with the `temporal-[db]-tool`
6. Run the `update-schema` command with the `temporal-[db]-tool` with a specific Visibility schema for that combination of database + schema version

The schemas (both [for initial setup](https://github.com/temporalio/temporal/blob/master/schema/postgresql/v96/temporal/schema.sql) and [for updating](https://github.com/temporalio/temporal/tree/master/schema/postgresql/v96/temporal/versioned)) are worth browsing if you are interested in understanding Temporal at a database schema level, but be warned that these tables are considered Temporal internals and should not be relied on.

If enabled, Temporal's Advanced Visibility Schema setup [works in a similar way](https://github.com/temporalio/temporal/tree/master/schema/elasticsearch/visibility) as described above.

## Temporal Server Setup

Once the schema is setup, the `setup_server` function is started in a background process. This function waits for Temporal server to get started which happens in a meantime in the main process (in `start-temporal.sh`). When Temporal server is started, this function does one more important thing that users may take for granted - **it registers the `default` namespace** using the standard `tctl --ns default namespace register` command.

- This is the namespace that most beginners are used to seeing when they start up Temporal inside docker-compose - because the `auto-setup.sh` script does it for them! Without this script running, Temporal Server doesn't have any namespaces registered. This can come as a surprise to some users who assume the default namespace is a necessary part of Temporal.
- All SDKs Workers and Clients, as well as Temporal Web, connect to the `default` namespace if not specified - most production deployments of Temporal will want to connect to specific namespaces. (try running [temporalite](https://github.com/DataDog/temporalite) without any namespaces to see how SDK code without namespaces specified stops working!)
- The script allows you to customize the name and retention period with the `DEFAULT_NAMESPACE` and `DEFAULT_NAMESPACE_RETENTION` env vars if you wish, but a production setup may want to register a number of namespaces differently.

The final part of the `auto-setup.sh` script also does registers some pre-defined custom search attributes (when running with Elasticsearch):

```tsx
tctl --auto_confirm admin cluster add-search-attributes \
          --name CustomKeywordField --type Keyword \
          --name CustomTextField --type Text \
          --name CustomIntField --type Int \
          --name CustomDatetimeField --type Datetime \
          --name CustomDoubleField --type Double \
          --name CustomBoolField --type Bool
```

These are intentionally generically named, for demo purposes (e.g. [for code samples](https://github.com/temporalio/samples-go/blob/77728cf7c38570898b2c90bf6eb0720c7f5fb30d/searchattributes/searchattributes_workflow.go#L56-L63)) and for ease of use (since they are already set up, you don't have to add them when you find you need them later). As you advance in your usage, you may wish to drop this step in favor of better named and specified attributes - just keep in mind that you are [limited to a maximum of 100](https://docs.temporal.io/kb/temporal-platform-limits-sheet).

## Closing Thoughts

Now that you have a better idea of where Temporal Server ends and optional tooling begins, you can hopefully better adapt Temporal to your preferred infrastructure and recovery/upgrade tooling, and have more confidence in debugging issues as you support Temporal at your company. We are also working on more one-click-deploy solutions to other vendors like [Render](https://github.com/sw-yx/temporal-render).

If you have followup questions, please feel free to ask on [Discourse](https://community.temporal.io) or [Slack](https://temporal.io/slack)!


148215905-ffb15e1a-75ba-4ddc-aa91-12d9e374f2d1

We often say that "Temporal Server is a simple Go binary", and it's true: Each release includes 4 docker images, but also zipped versions of our temporal-server binary.

How Temporal Auto-Setup Works

## Introduction

Workflows are based on the concept of capturing and expressing our business processes in a structured way. Workflow technologies are software components that promote the use of workflows for building understandable, explainable applications.

All workflow technologies provide two core capabilities:
1. Ways to define your workflows
2. A runtime that executes them

Additionally, most provide integrations with different frameworks, testing and debugging capabilities, visibility and observability features, etc.

In the first of two articles, we will give a general introduction to workflows and their importance. We will also look into different, currently-relevant approaches of defining workflows and provide their pros and cons. In the subsequent article, we will dive deeper into this subject and provide more insight into developing a real-life workflow solution using these different approaches.

## Defining workflows

Workflows can be defined in many different ways. One way that we are probably most familiar with is by just using a pen and paper. As we are planning out our day, we can write all of our tasks as a list with some defined order. This list then represents our workflow, or our structured business plan for the day. Let’s say that we write the following initial outline of our morning tasks. Note this is actually our first workflow definition.

![ofupug76zsdoq64rdkrn](//images.ctfassets.net/0uuz8ydxyd9p/4q3ZrnHLFbHiCv0ubx6gBz/95c8568861fe783d16b8bdcb61bcaf20/ofupug76zsdoq64rdkrn.png)
*Figure 1. Initial workflow*

Workflows typically have a set goal, the result we are trying to achieve. In our case, the goal is to get to work on time. Each task in our workflow can be seen as a sub-goal or task that is important to achieve in order to reach our goal.

Our initial workflow is not perfect, and can be improved. For example, we notice that in order to give us more time to eat breakfast, we could shower and brush our teeth at the same time. Furthermore, we can make our breakfast while making coffee and lunch, as well as read our newspaper while eating. We can update our workflow to look like this:

![80ea8yh5xqbl72uk0omk](//images.ctfassets.net/0uuz8ydxyd9p/5WUxYObUVSTbSOMjHjAg8p/b73315de0c6cb08c3a7049b632a5aea3/80ea8yh5xqbl72uk0omk.png)
*Figure 2. Revised workflow*

Next thing we need to realize is that this plan represents a “perfect” scenario. It does not for example deal with failures that we know could happen. We need to update our workflow to deal with predictable failures. Updating workflows is an iterative process. Updates can be based on plain common sense, but also from experience gained over time. One update we want to add to our current workflow is to check the traffic report before driving to work. If the traffic is bad, we will need to notify our boss that we will be late:
re 3. Further updates

![zn3zogzvwphkgbnr98fk](//images.ctfassets.net/0uuz8ydxyd9p/6MeOp7rcRYB8YznpCjPCgh/f9b22fdc3742f147a7d4117abaf67671/zn3zogzvwphkgbnr98fk.png)

*Figure 3. Further updates*

Notice that we used the adjectives “ok” and “bad” to describe traffic conditions. These adjectives correspond to what we think is ok or bad depending on things like the distance from our home to work, our current car condition, our driving abilities, etc. An important part of the way we define and think about workflows is very domain-specific, meaning that it is closely related to our own reasoning, how we process information, and our current circumstances.

One thing we realize is that no matter how bullet-proof we think our workflow is, it often is not. Uncertainty and having to deal with unexpected issues is part of life.
For example, let’s say we are making our breakfast burrito, it falls on the floor and our dog eats it. We can recover from this failure by making another one, while still maintaining the successful completion of our main goal. In the case we don’t have the ingredients any more to make another burrito, we can recover from this intermittent failure by making a sandwich instead.

Adding all possible intermittent failures to our workflow definitions is not feasible. There are just too many of them, some of which we cannot predict. Our workflow needs to have the capability however to recover from any such intermittent failures. One possible way to recover is to retry a task that failed, another is to be able to compensate for a task that failed with one or more other ones, all while still being able to reach our goal.

Defining workflows in our tech world is based on the same principles we used to describe using our simple morning routine workflow written on a piece of paper.
All the strategies we were able to apply to our morning routine workflow so far, such as
Iteration-based improvements, domain-specific terminology , dealing with known and unknown failures, recovering from failure, and the ability to adapt to changes, all apply to workflows targeting any domain.

## Why workflows, really

Over the last decade and more that I’ve been involved with workflow technologies, people often asked me to define what is a workflow, and why to use workflows.

The answer to “What is a workflow?” is kind of a tricky one. If you put ten people in the same room, you will get ten different answers and each one is going to be correct.
One answer that resonates to me more than others is that workflows are means to define a structured set of objectives or tasks needed to achieve a desired goal.

The answer to “Why use workflows?” (especially in the tech field), is very important to me on the other hand, and is the reason why I have been a workflow enthusiast for many years.
It’s true that we can write any sort of application without using workflows. However, by doing so, we are missing out. Workflows provide the most natural way to describe how our processes should work to both humans as well as computers.
Our workflow definition maps to how we think our applications should behave and give instructions to computers on how to execute them.

Unfortunately we cannot just scan our workflow definition that we wrote on a piece of paper and execute it. Workflow technologies, over the last years, have developed many different approaches to defining executable workflows. In the next sections we will talk about the most widely used ones.

## Common approaches for defining workflows

The two most widely used approaches for defining workflows are
* Domain-specific languages
* General-purpose programming languages

### Domain-specific languages

Domain-specific languages (DSLs) provide a higher level of abstraction compared to general-purpose programming languages. They also tend to target a very specific class of problems. The idea of domain-specific languages is to expose a smaller subset of functionality that can be used to write our workflows. This subset should be very specific to a single technology domain, and thus can be more efficient for defining workflows in that domain.
DSLs can thus be seen as specialized languages. Trying to solve a problem in one domain using a DSL that specializes in another is often inefficient or even impossible. Similarly, DSLs that attempt to specialize in multiple technology domains often lack functionality of those that specialize solely on one.

Domain-specific languages are typically exposed in three ways:
* Flowcharts
* Forms
* Markup

Both Flowchart-based and Form-based DSLs allow us to visually create our workflow definitions. They target non-technical businesses to allow them to share and describe workflows to other non-technical, as well as technical parties.

Flowchart and Form-based workflow solutions have been around for many years and are in most cases suited to represent workflows of small to medium complexities.

![s66vvrvrq3an44w0aicn](//images.ctfassets.net/0uuz8ydxyd9p/6qV6QV6M5XByaiZzvyD9hI/ae6215c041f5abd457615db1f7bb7743/s66vvrvrq3an44w0aicn.png)
*Figure 4. Sample visual workflow*

Markup-based DSLs allow us to use light-weight formats such as JSON or YAML to define our workflows. They too target specific technology domains so are well suited for workflows of small to medium complexities. Markup-based DSLs do not rely on a visual representation; however many tools exist to provide this feature. In addition, they can be embedded inside resource definitions of many different frameworks and container-based solutions which use the same markup, making them attractive for integration into existing tech solutions.

![lh8svxp3ah96wmpnv0n9](//images.ctfassets.net/0uuz8ydxyd9p/GHdMpvAaNr3gZNp9lQcEj/740fe75811bd6ed201e9e2e9d2551a78/lh8svxp3ah96wmpnv0n9.png)
*Figure 5. Sample markup-based DSL workflow*

It is important to mention that whether we use flowchart, form, or markup-based DSLs, these definitions are not executable by computers and require an additional translation step which converts them into executable programming language code.

#### Pros and Cons of defining workflows with DSLs

We already touched upon some benefits of defining workflows with DSLs such as domain-specialization, visualization, ability to be understood by both technical and non-technical users, as well as easy integration with existing frameworks and tools for markup-based DSLs.

Flowchart-based DSLs typically provide a drag-and-drop experience that is intuitive to many users. Markup-based DSLs can benefit from being easily integrated into existing tooling, such as many different editors and can benefit from their Intellisense features, as well have the ability to be visualized.
Markup-based DSLs also have the benefit of being fairly easy to create and can be customized to fit the end-users specific needs.

As mentioned, when writing workflows using DSLs we should focus on solving problems of small to medium-sized complexities.

DSLs often run into issues when trying to use them for complex workflow definitions.
For flowchart-based workflows, the visualization aspect can quickly lose its benefits and become hard to understand and reason over.

![lr5kkf8dy59hlitj79me](//images.ctfassets.net/0uuz8ydxyd9p/6mDm2KJRLYGS1j1oj5iqQh/adb19e529e94f38cca2aafffcf7620ad/lr5kkf8dy59hlitj79me.png)
*Figure 6. Example large flowchart-based workflow*

Another issue we can run into is the need to overuse programming language code that is embeddable into the visual workflow representations. Flowchart-based DSLs allow for embedding code as non-visual parameters of certain control-flow logic blocks. This code performs some aspects of the control-flow logic that cannot be expressed by visualization alone. For complex workflow definitions, we often need to add large amounts of code to define certain parts of our workflow logic that can lead to our visual model becoming misleading as to what our workflow is actually supposed to do.

![qo4aor9twlbn4itzlh3l](//images.ctfassets.net/0uuz8ydxyd9p/3s0YSs2vMiOtqYMeX6WiTX/bbf49ce2a8ac377829b3101053f24708/qo4aor9twlbn4itzlh3l.png)
*Figure 7. Code inside flow-chart-based DSL*

Markup-based DSLs experience similar issues when dealing with complex workflow definitions.
As they heavily rely on string-based references, defining complex workflow control-flow logic can become very hard to follow and understand. The same applies again for possible over-use of code, in the case of Markup-based DSLs, expression language syntax inside complex workflow definitions that can become hard to grasp.

One more thing that’s important to mention about DSLs is their benefit of being very domain-specific and ability to be used by both technical and non-technical users comes at a price.
DSLs often demand custom ecosystem support, namely tooling that has to be specifically created for that DSL (which are oftentimes proprietary). This is especially the case for flowchart-based DSLs.
Switching from one DSL to another to define our workflow solutions for different technology domains often leads us to having to not only learn a new DSL, but also having to get familiar with vastly different ways to test, debug, deploy, and interact with our workflows.

### General-purpose programming languages

General-purpose programming language workflow models target technical users (developers).
As the name implies, they also target a generic class of problems and can be leveraged to define workflows targeting a broad set of technology domains. For this reason they include a much wider set of features compared to those typically exposed by DSLs.

When writing our workflow definitions using code, we can take advantage of many and often all of the programming languages features, and just like generic code that we write, our workflow definitions can scale along with their complexity.

![5l9oer3kyyx395twh4ij](//images.ctfassets.net/0uuz8ydxyd9p/2JZ7Ym04kfN6VX1S7RBdEy/66f900d420124aca639198efae2c4937/5l9oer3kyyx395twh4ij.png)
*Figure 8. Sample programming-language based workflow*

#### Pros and Cons of defining workflows using programming languages

Programming languages are one of the most widely used and popular technology standards out there. Because of this popularity they have very large ecosystems and mature tooling support for things like documentation, testing, debugging, deployment and framework integration. Many IDEs and frameworks have support for multiple programming languages which makes switching out workflow definitions from one language to another a lot easier. Popular programming languages often have huge communities which leads to stable continuous language innovation, as well as quick turnarounds on improvements and bug fixes.

For developers, learning to write workflow definitions using programming languages comes naturally, as they are already familiar with coding and can easily integrate workflows as part of their existing and new development/architecture efforts.

Code-based workflow definitions do not appeal to non-technical users and cannot be easily used as means to describe business logic across different teams with wide ranges of technical knowledge levels.

Given that they are based on general-purpose programming languages, code-based workflow definitions can become complex to understand. Defining a common set of libraries and solutions for specific control-flow logic implementations, alongside the language's features such as reusability, orthogonality, and abstraction amongst many others, can often aid in lowering this complexity.

## Conclusion

I hope that this article has given you a good introduction to what workflows are and why you should use them. In addition, we went through the most common approaches to defining workflows and mentioned their pros and cons.

In the next article, we will dive deeper into this subject and provide more insight into developing a real-life workflow solution using these different approaches.

Stay tuned :) 


ricardo-gomez-angel-9AdeEdYB2yk-unsplash

Workflows are based on the concept of capturing and expressing our business processes in a structured way. 

Defining Workflows in Temporal

Tihomir Surdilovic

tihomir surdilovic

Defining Workflows

*Since this post was published, the TypeScript SDK has switched to using the [`vm` built-in Node module](https://nodejs.org/api/vm.html). However, the concepts described remain the same.*

## Introduction

[Temporal's TypeScript SDK](/typescript/introduction) makes heavy use of V8 isolates via the [isolated-vm npm package](https://www.npmjs.com/package/isolated-vm) to ensure your [Workflows are deterministic](/typescript/determinism). Each Workflow runs in an _isolate_ that prevents your Workflow code from directly accessing any logic that may break Workflow determinism, like reading from the file system or generating a random number. In this blog post, I'll cover why the TypeScript SDK V8 isolates, sketch out how the TypeScript SDK uses isolates, and describe what that means for how you write Workflows in TypeScript.

## Why Isolates?

The key idea behind Temporal is that each Workflow is stored in the Temporal Server as an [_event history_](/blog/temporal-tips-tricks-1/#event-history). Instead of storing the "current state" of each Workflow, Temporal instead stores the entire history of the Workflow. That includes the Workflow's initial state, all Signals the Workflow has received, and the result of all Activities the Workflow executed.

The [Temporal Server](https://docs.temporal.io/clusters/) can pause the Workflow and resume it on a different Worker. For example, the Workflow may be evicted from the [Worker's cache if the cache is out of space](https://docs.temporal.io/concepts/workflows/#faq). To restore the Workflow, The Temporal Server will replay the Workflow function from the beginning using the same Signals and Activity results. However, replaying event histories assumes that the Workflow is fully _deterministic_. That means the Workflow will be in the exact same state given the same initial state and same sequence of events.

There are a lot of potential sources of non-determinism. Some are obvious, some are more subtle. For example, suppose you tried to write a Workflow that sleeps for a conditional amount of time based on the current time.

```ts
import { sleep } from '@temporalio/workflow';

export const sleepWorkflow = () => ({
  return {
    async execute(): Promise<void> {
      const start = Date.now();
      // Sleep until the end of the current second
      await sleep(1000 - start.valueOf() % 1000);
    }
  }
})
```

If `sleepWorkflow()` ran in vanilla Node.js, it would **not** be deterministic because `Date.now()` is not deterministic. If the Temporal Server pauses the Workflow and later runs `execute()` to restart the Workflow, the Workflow will sleep for a different amount of time unless the Temporal Server happens to restart the Workflow at exactly the right time.

The Temporal TypeScript SDK runs your Workflow code in an isolate to remove sources of non-determinism. Running in an isolate means each Workflow execution has its own memory, so there's no way for Workflows to share a singleton or any other form of shared state. And the Temporal TypeScript SDK can replace commonly used non-deterministic built-ins, like the `Date` constructor, with Workflow-safe replacements. For example, suppose you wrote a Workflow that printed out the `Date` constructor as a string as shown below.

```ts
export const testWorkflow = () => ({
  return {
    async execute(): Promise<Date> {
      console.log(Date.toString());
      return new Date();
    }
  }
})
```

Running `testWorkflow()` would print something like what you see below. Notice that `Date()` is now a stub that pulls the current time from the Temporal TypeScript SDK's internal state, rather than the system time.

```
example 00363f1d-19c7-429f-aabb-6067b4df2bd2 > function () {
        return new OriginalDate(internals_1.state.now);
    }
```

## Getting Started with isolated-vm

The isolated-vm module allows you to create a new _isolate_, and execute a _script_ within the isolate with a given _context_. Below is a "Hello, World" example of running a script within an isolate in isolated-vm. For the purposes of this tutorial we'll use the sync versions of isolated-vm's `compileScript()`, `createContext()`, and `run()` functions for brevity.

```ts
import ivm from 'isolated-vm';

const code = `(function() { return 'Hello, Isolate!'; })()`;

const isolate = new ivm.Isolate({ memoryLimit: 8 /* MB */ });
const script = isolate.compileScriptSync(code);
const context = isolate.createContextSync();

// Prints "Hello, Isolate!"
console.log(script.runSync(context));
```

A [context](https://www.npmjs.com/package/isolated-vm#class-context-transferable) is an object that contains the global variables available to the script when it runs in the isolate. You can share global values between runs by re-using the same context as shown below.

```ts
import ivm from 'isolated-vm';

const code = `++count;`;

const isolate = new ivm.Isolate({ memoryLimit: 8 /* MB */ });
const script = isolate.compileScriptSync(code);
const context = isolate.createContextSync();

context.evalSync('let count = 0;');

console.log(script.runSync(context)); // Prints "1"
console.log(script.runSync(context)); // Prints "2"
```

However, by creating a new context for every run, you can ensure that each `runSync()` call runs in an isolated environment, with no ability to interact with other `runSync()` calls. That's how the [Temporal TypeScript SDK ensures Workflows can't interfere with each other](https://github.com/temporalio/sdk-typescript/blob/5a0f780b9cb4c0dae265c08ca99fbc1f58c4ab83/packages/worker/src/isolate-context-provider.ts#L32-L40).

How does the Temporal TypeScript SDK handle overwriting the `Date()` constructor? The Temporal TypeScript SDK compiles your Workflows with [Webpack](https://www.npmjs.com/package/webpack) and creates an [entry script that overrides `Date()` before your Workflow starts](https://github.com/temporalio/sdk-typescript/blob/5a0f780b9cb4c0dae265c08ca99fbc1f58c4ab83/packages/worker/src/isolate-builder.ts#L89-L125). Webpack also bundles dependencies, which means your Workflows can use `import` statements.

## Takeaways for Writing Workflows in TypeScript

The Temporal TypeScript SDK runs Workflows in a clean v8 runtime that has several globals stubbed out, including [`Math.random()`, `Date()`, `setTimeout()`, and `clearTimeout()`](https://github.com/temporalio/sdk-typescript/blob/004c2846fe4e4312eb2c424da477bc0c280d6c48/packages/workflow/src/worker-interface.ts). In particular, accessing `WeakRef()`, and `FinalizationRegistry` throw errors, because garbage collection is non-deterministic. The Temporal TypeScript SDK also explicitly prevents you from accessing several Node.js APIs, including `fs` and `http`. For example, if you try to import `fs` in a Workflow, your worker will fail with the below error message at startup.

```
Module not found: Error: Can't resolve 'fs' in '/path/to/lib/workflows'
```

**Because of this, do not use any npm modules that read from the file system or make HTTP requests in your Workflows.**
Furthermore, because these npm modules are unlikely to check for Temporal's runtime environment, they will most likely crash with hard-to-read error messages. For example, suppose you try to [make a GET request with Axios](https://masteringjs.io/tutorials/axios/get) from a Workflow as shown below.

```ts
import axios from 'axios';

// Don't make non-deterministic calls inside Workflows
export const httpWorkflow = () => {
  return {
    async execute(): Promise<void> {
      return await axios.get('http://httpbin.org/get');
    }
  }
}
```

The `await axios.get()` call will throw a `TypeError: adapter is not a function` error, because Axios doesn't recognize the Temporal Workflow isolate runtime.

If you need to use an npm module that interacts with the outside world, like Axios or [Mongoose](https://mongoosejs.com/), you should create an Activity that uses the npm module. Activities run in the normal Node.js environment, **not** an isolate.

```ts
import axios from 'axios';

// Do place all side effects in Activities - Failures will be retried, use idempotency keys if necessary
export async function makeHTTPRequest(): Promise<void> {
  return await axios.get('http://httpbin.org/get');
}
```

## Moving On

The Temporal TypeScript SDK uses isolated-vm to ensure there's no way for developers to violate Workflow determinism. Your Workflow will throw an error if you intentionally or unintentionally write some non-deterministic code. This makes the TypeScript SDK different from some of Temporal's other SDKs, like the [Go SDK](https://docs.temporal.io/go/workflows#how-to-write-workflow-code), which currently rely on developers to avoid using language built-ins that violate determinism. Workflow determinism is a critical assumption in Temporal, so making it impossible to accidentally break Workflow determinism is a major benefit of the Temporal TypeScript SDK.


praewthida-k-5Ui5glVjUAQ-unsplash

Temporal's TypeScript SDK makes heavy use of V8 isolates via the isolated-vm npm package to ensure your Workflows are deterministic. 

Introduction to isolated-vm in TypeScript


In most discussions of Temporal use cases, we go straight to talking about "long running" work. In the past, I typically associated that sort of work with "use if I need to do video processing". But I was wrong.

**"Long running" usecases are both shorter *and* longer than you think.**

## Context

If you surveyed most developers for what a "long running job" typically means to them, they would usually point to a range of intervals from 10 minutes to 6 hours, based on their background.

A common shorthand we've used in the past is "anything beyond request/response". You can use AWS Lambda's default timeout of 3 seconds as a boundary between "request/response" and "long running". But even though you can extend that timeout to 15 minutes, would you really want to?

Perhaps "long running" isn't really about some arbitrary cutoff in time.

## "Long Running" can be really short

[Box uses Temporal](/blog/temporal-a-central-brain-for-box) for orchestrating file update operations. Although this can take hours for large transfer, the vast majority of these feel instantaneous to users. We ideally want one solution to scale from the smallest to largest usecases with no more visible latency than absolutely necessary. Box uses Temporal more for transactional and reliability guarantees around microservice orchestration, and the words "Long Running" were never even mentioned.

Workflow engines that rely on a centralized scheduler [have a confusing floor to their latency](https://stackoverflow.com/questions/49902599/airflow-latency-between-tasks) and a poor fit for cases where you want the work to be done **as soon as possible** rather than once a day. It *could* be long running, but mostly it's not, and the programming model needs to scale with your users rather than swap out systems based on some arbitrary threshold.

**Under the hood, Temporal accomplishes fast responses with [long polling](https://ably.com/topic/long-polling)**. Workers listen to Task Queues inside Temporal Server, which hold the HTTP connection open until the next Command is enqueued. While Temporal is not suited for truly realtime needs like gaming, this lets Temporal workers respond quickly to state changes and is much less resource intensive than using web sockets.

> Note: this is also why we strongly recommend [monitoring ScheduleToStart latency](https://docs.temporal.io/server/production-deployment/#scaling-and-metrics) for production deployments, as a spike in this latency metric is your first indication of a spike in traffic or problem with your Workers or networking.

## "Long Running" can be infinitely long

[Checkr uses Temporal](/case-studies/how-temporal-simplified-checkr-workflows) for coordinating background checks. This is a multi-staged process with a *huge* range in processing times, ranging from pinging a database search API, to dispatching a court researcher to a court house, followed by analyzing each record and potentially escalating to manual QA. The whole process could take days.

The problem with hand-rolled orchestration systems built out of queues and databases is that the longer a job runs, **the more moving parts, the more likely you will find which of the [fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing) you have forgotten**, whether it is failing to persist some state, instrumenting for observability and debuggability, or even rolling out changes with production infrastructure mid-flight.

Temporal solves this by persisting [event histories](/workflows/#event-history) as a source of truth, solving for both observability and reliability in one fell swoop. While there is a [soft limit](/kb/temporal-platform-limits-sheet) of 10,000 events in a single Workflow Execution, you can pick a natural cutoff point to snapshot state and start a new Workflow Execution with our [ContinueAsNew](/typescript/workflows/#why-continueasnew-is-needed) API.

In practice, this means you can write **infinitely long running Workflows.** Since Temporal makes state easy to store and fully auditable, this has the potential to change your programming model completely, by allowing you to model every user as a long lived Entity in your system.

> For example, you could use this for e-commerce: coordinating actions like loyalty rewards, subscription charges, and reminder emails over the entire history of their relationship with you.

## It's really about Uncertainty

The notion that "Temporal is for long running work" is more subtle than appears at first glance. If your microservices and third party APIs were always reliable and always responded quickly, you wouldn't need Temporal. If you didn't care about modeling and testing full journeys with code, from a multi-stage background check, to the entire history of a customer from signup to churn, then you could keep piecing them together on an ad-hoc basis.

But it's when you're *unsure* about parts of your distributed system that you need Temporal. **When you cross network boundaries, when you need to scale horizontally, or when you need to coordinate migrations**, you start needing a solution that separates these mechanical uncertainties from the business logic.

Most engineers have learned to deal with uncertainty piecemeal, painful incident after painful incident. We've assembled centuries of distributed systems experience to solve it for you.


hulki-okan-tabak-x3kQTL7yw30-unsplash

In most discussions of Temporal use cases, we go straight to talking about "long running" work. In the past, I typically associated that sort of work with "use if I need to do video processing". But I was wrong.

What does "Long Running" really mean?

Discussions about Temporal often get into very high-minded distributed systems debates, but sometimes it can be helpful to swap small tips that can have a huge impact on developer productivity!

Here's a small collection of 4 tips we put together for you.

## Auto-Restarting Workers

When you write Activity or Workflow code, you often have to kill and restart the Workers that host them, so as to pick up the changes. Usually this is a manual `Ctrl+C` + rerun process. However, you can set up your workers to reload on every *file save*. Since Workers are stateless and Workflows are tolerant to Workers going down, this works just fine.

There are many ways to achieve this. You can use [`gow`](https://github.com/mitranim/gow) as a drop in replacement for the go binary that reloads upon save.  `nodemon` from the JavaScript ecosystem is a nice fully cross-platform binary that watches files and executes scripts.

```bash
# using gow
go install github.com/mitranim/gow@latest
gow run path/to/worker.go

# using nodemon
npm i -g nodemon
nodemon --watch './**/*.go' --signal SIGTERM --exec 'go' run path/to/worker.go
alias gow="nodemon --watch './**/*.go' --signal SIGTERM --exec 'go' run " # alias for shortcut
```

Obviously constantly rerunning a command on every file save may be a problem in some cases, so use your best judgement.

## Code Snippets

This is a generally applicable developer productivity tip, but there will always be some amount of unavoidable boilerplate in code and Temporal is no exception. Roll your commonly used code into reusable snippets, and take advantage of modern snippet features like [Tab Stops, Placeholders, and Transformations](https://macromates.com/manual/en/snippets#tab_stops).

```go
// @prefix newclient
// @description add new client

// The client is a heavyweight object that should be created once
// options https://pkg.go.dev/go.temporal.io/internal#ClientOptions
${0:newClient}, err := client.NewClient(client.Options{})
if err != nil {
	log.Fatalln("Unable to create client", err)
}
defer ${0:newClient}.Close()
```

You can even leave comments to relevant documentation, like this:

```go
// @prefix RetryPolicy
// @description simple retry policy

// https://docs.temporal.io/concepts/what-is-a-retry-policy
    RetryPolicy: 			&temporal.RetryPolicy{
			InitialInterval:    time.Second,
			BackoffCoefficient: 2,
			MaximumInterval:    time.Minute * 10,
			MaximumAttempts:    5,
		},
```

You can look up how to do this in [TextMate](https://macromates.com/manual/en/snippets), [Atom](https://flight-manual.atom.io/using-atom/sections/snippets/), and [Sublime Text](http://docs.sublimetext.info/en/latest/extensibility/snippets.html).

In VS Code, you should try [the Easy Snippet extension](https://marketplace.visualstudio.com/items?itemName=inu1255.easy-snippet) to make writing [VS Code's User Defined Snippets](https://code.visualstudio.com/docs/editor/userdefinedsnippets) easier. Here's a [short video](https://twitter.com/swyx/status/1420772267772968960) demonstrating usage of Easy Snippet.

## Stack Trace

One of the lesser known but extremely useful features of [Temporal Web](https://docs.temporal.io/web-ui) is the ability to view stack traces.

![image](https://user-images.githubusercontent.com/6764957/127705775-8de7c70d-5d89-4b1b-8f4a-8dcd9ccb70f7.png)

This helps greatly with debugging and understanding how Temporal deals with asynchronous work and is able to resume workflows on a different machine.

## Event History

You can export the event history via [`tctl`](https://docs.temporal.io/tctl) or the [Web UI](https://docs.temporal.io/web-ui):

```bash
tctl workflow show \
  -w myWorkflowId \
  -r 866ae14c-88cf-4f1e-980f-571e031d71b0
```

This is not just good for recordkeeping, but also great for taking a failing production workflow and debug it locally, exactly as it was executed.

[Video: Workflow versioning with Temporal](https://www.youtube.com/watch?v=kkP899WxgzY)

Before a migration, you can [replay event histories in a test](https://github.com/tsurdilo/temporal-versioning-go/blob/033e04a75049c4bea79908e0619332360c3cddc3/tests/replay_test.go), to verify that you have written backward compatible code:

```go
func (s *replayTestSuite) TestReplayFromInitialVersion() {
	replayer := worker.NewWorkflowReplayer()

	replayer.RegisterWorkflow(workflow.CustomerWorkflow)

	err := replayer.ReplayWorkflowHistoryFromJSONFile(nil, "initversionhistory.json")
	require.NoError(s.T(), err)
}
```
To learn more, you can see [how Temporal tests our Workflow Replay feature](https://github.com/temporalio/sdk-go/tree/6580cbe0aa41a8b515791f95c2c15bb37db1dab1/test/replaytests) since we are open source!

## More tips?

If you have found Temporal specific or general developer productivity tips that Temporal developers might use, let us know, we'd love to do another version of this roundup!


Discussions about Temporal often get into very high-minded distributed systems debates, but sometimes it can be helpful to swap small tips that can have a huge impact on developer productivity!

Tips and Tricks for Temporal Developer Productivity

In this post, we’ll dive into a media processing problem that requires high reliability and fault tolerance, and cover how to build a robust solution to this problem using Temporal.

Media processing workflows

Engineering @ Stripe

Nir Padmanabhan

56231878


One benefit of moving business logic to Temporal is how Temporal implements retries and timeouts for Activities in a standardized way. This has the effect of adding a reliability layer atop unreliable Activities and Workers, in a durable and scalable fashion. However, understanding the terminology can be a bit intimidating at first glance. 

This post (together with the embedded talk) aims to give you a solid mental model on what each Activity timeout does and when to use it.

> Note: You can also set [Workflow timeouts](https://docs.temporal.io/concepts/workflows#timeout-settings) and [Retry Policies](https://docs.temporal.io/concepts/what-is-a-retry-policy) you can set. This post deals only with *Activity* timeouts.

## Talk version: whiteboard session

You can watch this 18 minute talk where our CEO Maxim Fateev explains the [4 Types of Activity timeouts in Temporal](https://www.youtube.com/watch?v=JK7WLK3ZSu8).

The rest of this post summarizes the main points covered in this whiteboard session, for those who prefer a written version.

## Written version: TL;DR

![122424123-0ab1e300-cfc1-11eb-82c5-cf8856a87242](//images.ctfassets.net/0uuz8ydxyd9p/qeHHAfhow01sZ2BXByG8q/bc9913b65ce1cf99a19521a19a78e5bf/122424123-0ab1e300-cfc1-11eb-82c5-cf8856a87242.png)

Temporal has four timeouts — two that are commonly used, and two that are useful only in specific cases:

- Schedule-To-Close: Limits the maximum execution time including retries.
- Start-To-Close: Limits the maximum execution time of a single execution. **We recommend ALWAYS setting this!**
- Heartbeat: Limits the maximum time between Heartbeats. *For long running Activities*, enables a quicker response when a Heartbeat fails to be recorded.
- Schedule-To-Start: Limits the maximum time that an Activity Task can sit in a Task Queue. Mainly to identify whether a Worker is down or for Task routing. **This is rarely needed!**

You can find the precise APIs in the reference documentation for each SDK: [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) and [Go](https://pkg.go.dev/go.temporal.io/sdk@v1.7.0/internal#ActivityOptions).

## Lifecycle of an Activity

To really understand how timeouts work, we should understand the typical lifecycle of an Activity as it journeys through the various parts of the system.

### Step 1 - Workflow Worker

An activity `SimpleActivity` is first invoked inside a Workflow Worker on Task Queue `sampleTaskQueue`. The precise method of invocation differs by SDK, and timeouts are also specified up front as part of Activity options:

__Example in Go__

```go
ao := workflow.ActivityOptions{
        TaskQueue:               "sampleTaskQueue",
        ScheduleToCloseTimeout: time.Second * 500,
        // ScheduleToStartTimeout: time.Second * 60, // usually not needed! see below
        StartToCloseTimeout:    time.Second * 60,
        HeartbeatTimeout:       time.Second * 10,
        WaitForCancellation:    false,
}
ctx = workflow.WithActivityOptions(ctx, ao)

var result string
err := workflow.ExecuteActivity(ctx, SimpleActivity, value).Get(ctx, &result)
if err != nil {
        return err
}
```

__Example in Java__

```java
@WorkflowInterface
public interface SimpleWorkflow {
    @WorkflowMethod
    String simpleWorkflowMethod(String someArg);
}

public static class SimpleWorkflowImpl implements SimpleWorkflow {
    private final SimpleActivities activities =
        Workflow.newActivityStub(
            SimpleActivities.class,
            ActivityOptions.newBuilder()
              .setScheduleToCloseTimeout(Duration.ofSeconds(500))
              .setStartToCloseTimeout(Duration.ofSeconds(60))
              .setHeartbeatTimeout(Duration.ofSeconds(10))
              .build());

    @Override
    public String simpleWorkflowMethod(String name) {
      return activities.simpleActivity("Hello", name);
    }
}
```

Behind the scenes, the SDK transforms this into a `ScheduleActivity` Command, which is sent to the Temporal Server. This Command includes various metadata, including the activity type (`SimpleActivity`), activity task queue (`sampleTaskQueue`), and activity ID. If a `RetryPolicy` is not specified, Temporal uses [the default Retry Policy](https://docs.temporal.io/concepts/workflows#retry-policy).

### Step 2 - Temporal Server

Receiving the Command, Temporal Server adds an Activity Task to the `sampleTaskQueue` Activity Task Queue. We explain details of how Temporal accomplishes this in [Designing A Workflow Engine](https://docs.temporal.io/blog/workflow-engine-principles/).

> The Activity Execution is now in a `SCHEDULED` state.

### Step 3 - Activity Worker 

An Activity Worker that has been polling the `sampleTaskQueue` Activity Task Queue picks up the Activity Task and begins execution.

> The Activity Execution is now in a `STARTED` state.

### Step 4 - Temporal Server

Once the Activity Execution finishes successfully, the Activity Worker sends a `CompleteActivityTask` message (together with the result of the Activity Execution) to Temporal Server, which now gives control back to the Workflow Worker to continue to the next line of code and repeat the process.

> The Activity Execution is now in a `CLOSED` state.

We have just described the "Happy Path" of an Activity Execution.
However, what happens when an Activity Worker crashes midway through an execution?

## Start-To-Close Timeout

We use the Start-To-Close timeout to control the maximum amount of time **a single Activity Execution** can take. We recommend **always** setting this timeout.

![122290108-fff64000-cf25-11eb-92b3-0533e41c3fee](//images.ctfassets.net/0uuz8ydxyd9p/7snGs8rLhy0iCqGtKNOrRq/0289f851e69b94722ccdbfdc64e2737c/122290108-fff64000-cf25-11eb-92b3-0533e41c3fee.png)

The classic example for why the Start-To-Close timeout is relevant, is when an Activity Task has been picked up from the Activity Task Queue (`STARTED` state) but the Worker crashes after that (so the Activity Execution never reaches `CLOSED` state). 

- Without a timeout configured, Temporal would never proactively timeout this Activity Execution to initiate a retry. The Activity Execution becomes "stuck" and the end user would experience an indefinite delay of their Workflow Execution with no feedback.
- With the timeout configured, Temporal registers an `ActivityTaskTimedOut` event internally which triggers the Server to attempt a retry based on the Activity Execution's `RetryPolicy:`
    - The Server adds the Activity Task to the Activity Task Queue again.
    - The Server increments the attempt count in the Workflow Execution's mutable state.
    - The Activity Task is picked up again by an Activity Worker.
    - In the Temporal Server, the Start-To-Close timer is reset and will fire again if this second attempt fails.

The tricky part of setting Start-To-Close is that it needs to be set longer than the maximum *possible* Activity Execution, since you want to avoid premature timeouts for Activity Executions that genuinely take that long. 
Practically, if an Activity Execution can take anywhere from 5 minutes to 5 hours, you need to set Start-To-Close to be longer than 5 hours.
If you have a long running Activity Execution like that, then we suggest using Heartbeat timeouts.

## Schedule-To-Close Timeout

The Schedule-To-Close Timeout is used to control the overall maximum amount of time allowed for an Activity Execution, including all retries.
This timeout only makes sense if the Activity Execution has a `RetryPolicy` with `MaximumAttempts > 1`.

![122290183-0dabc580-cf26-11eb-913d-3dc74d5eb55f](//images.ctfassets.net/0uuz8ydxyd9p/4BrwBIxb3q2KkolnU73G3d/8a667d9b619d72487914a9c4b1d5ce2b/122290183-0dabc580-cf26-11eb-913d-3dc74d5eb55f.png)

If you let it, Temporal will retry a failing Activity Execution for up to 10 years! (with exponential backoff up to a defined maximum interval)
Most Temporal developers will want to fine-tune retries to balance the user experience against the unreliability of the Activity Execution.

While you can control intervals between retries and maximum number of retries in the `RetryPolicy`, the Schedule-To-Close Timeout is the best way to control retries based on the overall *time that has elapsed*.
We recommend using the Schedule-To-Close Timeout to limit retries rather than tweaking the number of Maximum Attempts, because that more closely matches the desired user experience in the majority of cases.

## Heartbeat Timeout

For long running Activity Executions, we recommend recording Heartbeats to create more frequent ping-backs from the Activity Worker to the Temporal Server. 
Then, set a Heartbeat Timeout to fail the Activity Execution when the Temporal doesn't receive a Heartbeat from your Activity Execution at the expected frequency.

![122424268-287f4800-cfc1-11eb-8417-f52743347afc](//images.ctfassets.net/0uuz8ydxyd9p/6CSLJVfVeIi0tldRAYd05G/610d84f98fce305489d44fe658c77ea4/122424268-287f4800-cfc1-11eb-8417-f52743347afc.png)

Heartbeats must be recorded from Activity Definitions (Activity code) using SDK APIs:

__Example in Go__

```go
progress := 0
for hasWork {
    // Send heartbeat message to the server.
    activity.RecordHeartbeat(ctx, progress)
    // Do some work.
    ...
    progress++
}
```

__Example in Java__

```java
while ((read = inputStream.read(bytes)) != -1) {
  totalRead += read;
  f.write(bytes, 0, read);
  // Let the Server know about the download progress.
  Activity.getExecutionContext().heartbeat(totalRead);
}
```

Setting a Heartbeat Timeout enables you to retry Activity Executions more quickly (e.g. the next minute after a Heartbeat is missed) than the Start-To-Close Timeout, which must be set to as long as the longest possible Activity Execution (e.g. five hours later when we are sure the Activity Execution should have been completed).

There are some minor nuances to Heartbeats that may be of interest:

- You can freely record Heartbeats as often as you want (e.g. once a minute, or every time a loop iterates). The SDKs throttle the Heartbeats that get sent back to the Server anyway.
- If a Heartbeat Timeout isn't set and the Activity Execution tries to record a Heartbeat, nothing will be recorded since that information will never be used.

## Schedule-To-Start Timeout

The Schedule-To-Start Timeout sets a limit on the amount of time that an Activity Task can sit in a Task Queue. We recommend that most users monitor the `temporal_activity_schedule_to_start_latency` metric and set alerts for that as a [production scaling](https://docs.temporal.io/server/production-deployment/#faq-autoscaling-workers-based-on-task-queue-load) metric, rather than setting an explicit timeout for it.

![122290279-287e3a00-cf26-11eb-8dd6-3133016a0bd9](//images.ctfassets.net/0uuz8ydxyd9p/4sBCBJXm4VBqpL5OnTEVxG/9e18181745c636de8d1ff43baa71b36b/122290279-287e3a00-cf26-11eb-8dd6-3133016a0bd9.png)

As a queue timeout, `ScheduleToStart` is unique in that it **doesn't result in a retry** — all a retry would do is pop the activity right back on to the same queue!

The Schedule-To-Start Timeout is most useful when you have a concrete plan to reroute an Activity Task to a different Task Queue, if a given Task Queue is not draining at an expected rate.
You can also reschedule a whole set of other Activity Executions or do other compensation logic based on this timeout.
This is a powerful feature for building ultra-reliable systems, however most users will not need this since you can horizontally scale the number of Workers easily.

Generally, issues with Schedule-To-Start are better addressed by scaling Activity Workers accordingly, rather than by adding timeouts.

## Putting it all together - a recruiting example

We've recently found the terminology of timeouts useful even internally for our recruiting and realized that this could be a relatable analogy for most people.

As [we are hiring heavily at Temporal](https://temporal.io/careers), there is a lot of interview scheduling going on intermixed with our regular day jobs. You could model the end-to-end hiring process for a candidate as a single "Workflow" with multiple "Activities": sourcing, interviewing, making a decision, and then the offer process. 

Specifically for the "Activity" of interviewing we've encountered some pain points:

- Some interviews were scheduled for 90 minutes, which we found to be way too long
- Some interviews had no-shows for any number of reasons ranging from miscommunication to Life™ getting in the way, and we were unclear on when to call an end to the interview and try to reschedule
- Sometimes a process would just drag out for weeks and weeks, leaving other candidates waiting.

To resolve this, we could think about setting some timeout policies (for clarity, none of these are real numbers):

- A Start-To-Close Timeout of 45 minutes so we don't spend too much time per interview.
- A Heartbeat Timeout of 10 minutes to see if we should cancel on no-shows.
- A Schedule-To-Close Timeout of 4 weeks to limit the length of time we spend accommodating other people's schedules vs our own.

![122290324-359b2900-cf26-11eb-93a6-5027fc98593b](//images.ctfassets.net/0uuz8ydxyd9p/1M8LmMf11gLaFi8mPw9uUL/6b533fe134d69ae47ec01691f597d179/122290324-359b2900-cf26-11eb-93a6-5027fc98593b.png)

Should we set a Schedule-To-Start timeout? You could imagine candidates sitting a queue waiting to be interviewed, with not enough interviewer "Workers" to process them. A timeout here wouldn't help much, because there's no other queue to put them on. Better to set up monitoring and alerting on Schedule-To-Start latency, and scale up workers accordingly as needed (autoscaling is [currently not possible](https://docs.temporal.io/server/production-deployment/#faq-autoscaling-workers-based-on-task-queue-load)).


One benefit of moving business logic to Temporal is how Temporal implements retries and timeouts for Activities in a standardized way. 

The 4 Types of Activity timeouts

I'm Ryland from Temporal - an MIT OSS platform for building highly reliable distributed applications. We pride ourselves on the reliability and consistency Temporal provides to our users mission critical applications. As a testament to that, in the last two years a data-loss bug has never reached production. Until last week.

## What happened?

Before we get into the details of the bug, I want to talk about how everything played out.

A few months ago, we started seeing strange behavior from the canary clusters we run for our cloud service. Some workflows were occasionally being created without an initial event, which left them in a highly inconsistent (ie: corrupted) state.

It's worth noting that Temporal is an incredibly well tested piece of technology, boasting thousands of unit, integration and scale tests. At the time the bug started appearing, we had also just moved to a new persistence vendor. This is why, we initially assumed that the bug was likely happening at the persistence level. So we reached out to the team who runs our persistence towards the end of December:

![slack-message-samar](//images.ctfassets.net/0uuz8ydxyd9p/5P3qimFsLhTlQW6OM2jKq2/8e069fd2278f65d717d4fb6a1ee3be67/slack-message-samar.png)

*A concrete example. At this point we understood exactly what was wrong with the workflow. We just didn't know why.*

At first, our persistence vendor agreed with the assessment. There had been some notable memory issues with the clusters running our persistence that seemed to directly correlate with the corruption we experienced. They very kindly quickly (and kindly) started investigating things on their side to root cause the out of memory issues.

![slack-message-samar-2](//images.ctfassets.net/0uuz8ydxyd9p/79L9ubkqUTyrwPR7uN8Ozv/16bb1e72cba31e72b449fd0a045ee23c/slack-message-samar-2.png)

*We discovered that there was a curious set of OOM (Out of memory) issues that corresponded closely with the corruptions we were experiencing with workflows.*

After a short time, they got back to us with what seemed like concrete progress. Their investigation had indeed found a memory issue with the cluster. At this point all we needed to do was validate that the memory issues always correlated with our workflow corruptions and we were golden.

![slack-message-samar-3](//images.ctfassets.net/0uuz8ydxyd9p/46korJDbvG9vJHee1dDPr1/54283b01d3dc57874ee1f8353e8ffc81/slack-message-samar-3.png)

*First formal identification of the OOM issue and how it correlates with the Temporal specific bugs. To validate that it was the root cause we started trying to correlate OOMs with Temporal corruptions.*

They immediately began working on a solution for the OOM issue. Within a week they had produced fixes and deployed those patches to the clusters running our persistence. We were very happy to get this issue behind us and get back to working on our product. Unfortunately this was not the end of the issues...

![slack-message-samar-4](//images.ctfassets.net/0uuz8ydxyd9p/2LIa2IKOdjRqZsPTPgoeD1/b3d9276ddd38a158f2018c6c6df48688/slack-message-samar-4.png)

*Our team converging with vendor on the OOM issues. At this point it seemed that fixing the OOM was the root cause and therefore fixing it would address our issues.*

Initially we had doubts about how much the fixes solved the memory issues. But it started to become clearer and clearer that something else was going on.

![slack-message-samar-5](//images.ctfassets.net/0uuz8ydxyd9p/7yHaAuW5Y6Sak2wvHA6zgr/48cdbb53671c9ff9d8ca6161599fe329/slack-message-samar-5.png)

*Our vendor explaining that the OOM fix was in place and functioning correctly. Temporal was still experiencing issues at this point which meant it was bigger than OOM.*

There was still the open question of why OOM's correlated so highly with the workflow corruptions, but we had to accept that it wasn't the full story. Our team began putting all of their energy into combing through our source code for potential issues. After a prolonged period of stress and frustration, one of the awesome engineers from our persistence vendor found the root cause:

![slack-message-samar-6](//images.ctfassets.net/0uuz8ydxyd9p/74QY3yCZKZsk43jMJAVHui/9ebfe0429568a18ad871dedd41add0cd/slack-message-samar-6.png)

*An amazing find by the engineer working for our vendor. We are super appreciative for their incredible dedication to helping us.*

Yes, you read it correctly. This entire mess was caused by a Golang variable shadowing issue in one small part of our persistence code.

This explains why the workflows were corrupted, but why was the issue so hard to reproduce and why did it only happen with Cassandra? The answer is complex, but the TL;DR; is that workflows would only become corrupted when Cassandra returns a specific error from a failed transaction. 99.9% of the time, Cassandra is operating correctly and these errors are never returned. Therefore 99.9% of the time workflows aren't corrupted. But in cases where Cassandra is experiencing instability (ie: ring changes, or OOM), it is possible for these errors to be returned thereby triggering the bug which was in Temporals persistence code. While we still don't fully understand the exhaustive set of conditions required for this to happen, we knew enough to reproduce the issue reliably.

Probably worth mentioning how we verified the fix. We asked the persistence provider to give us a cluster with OOM issue so we can run Temporal with the fix with frequent OOMs. We ran a stress workload for a 3 days with OOM every 1 hour and able to verified that no workflow executions were left in corrupted state.

## We mess up

Before I get into the details of the bug at a code level, I want to talk about how we responded very poorly to the situation. Once we confirmed the nature of the bug, we produced a patch and deployed it to our canaries. We then asked our persistence vendor to spin up a cluster that was experiencing the OOM issue. On the OOM cluster we ran a 3 day stress test with the patched version of Temporal. At the end of the test zero workflows were left in a corrupted state giving us confidence in the stability of the fix. We also scheduled some time to discuss how we want to handle this bug in regards to OSS releases. At this point we made a clear error in judgement. During the team meeting regarding the bug, we discussed whether we should explicitly announce the bug to our users. In hindsight, the fact that we even discussed "whether" instead of "how" was a problem in itself. If you hadn't already guessed, we chose to include the fix in the next release but not announce the bug itself. For those curious about the rationale, it was mixed:

The bug only affects Cassandra users and we do not know of any production Cassandra users of Temporal (we still believe this was true at the time)
The bug does not happen in 99% of cases, only when the underlying Cassandra cluster is experiencing instability. Specifically when someone is trying to scale the underlying Cassandra cluster.
The bug does not affect existing workflows and only applies to ones that are newly created. It's cringeworthy to think about now, but we even questioned whether this is "really dataloss".
At the end of the day, even if the rationale had been good (it wasn't) it goes against the core values of our company and team. We believe transparency is a fundamental and inseparable part of Temporal and we were the opposite of transparent here.

Unfortunately, the story does not end at this point. We proceeded with releasing version 1.6.0 that included the fix, but failed to adequately call out the severity of the bug.

![slack-message-ryland](//images.ctfassets.net/0uuz8ydxyd9p/4bwmk1ucfGqzjXnhQ1MXuE/05f318ab9ce5b8419535aaa15c1770c8/slack-message-ryland.png)

## Someone pays the price

Last week, about 1 month after releasing 1.6.0 we received some very concerned messages from one of the largest OSS Temporal users. They had semi-recently moved into production and were noticing some errors which looked very similar to the corruption issues caused by the bug. After a bit of debugging, we confirmed that it was the same shadowing bug causing the issues. While we had properly fixed it in 1.6.0, they had not yet fully upgraded their Temporal deployment and therefore still had the broken code path. To be clear, the user should not be blamed at any level. If we had been more proactive about messaging in the first place, they would have definitely upgraded and never run into the issue. While we had made a lot of regrettable choices at this point, we decided to stop that trend the next day. An announcement was made to the entire community about the bug, and we backported the fix to all older versions.

That's nice and all, but it doesn't say anything about the future. Why won't this happen again? The short answer is that we are absolutely dedicated to this never happening again. We have a few actionable steps to make sure that ends up being the case:

- We will be adding tests to validate how the system behaves when Cassandra clusters(and potentially other persistence) are changed. In general we will be hardening our tests to properly handle dependency-level failures. [https://github.com/temporalio/temporal/issues/1153](https://github.com/temporalio/temporal/issues/1153)
- A linter will be added to catch the variable shadowing issue. We will also proactively re-evaluate our linting to determine if other lint rules can be added to reduce likelihood of bugs.
- Improvements will be made so that it will be easier to remove (or fix if possible) corrupted workflows gracefully. As of now they put extra load on the system and add noise to the logs. [https://github.com/temporalio/temporal/pull/1312](https://github.com/temporalio/temporal/pull/1312)
- Communication will be improved. In addition to being more transparent and vocal about such bugs over existing communication mediums, we will also add additional broadcast mechanisms to alert users of critical bugs ASAP. Specifically we will augment the version update mechanism of Temporal Web so it can alert users if they are running on Temporal versions with known issues. [https://github.com/temporalio/web/commit/45a7e9ccc36a7703f3270a664e40048cf077aeee](https://github.com/temporalio/web/commit/45a7e9ccc36a7703f3270a664e40048cf077aeee)

For those curious about why the bug was encountered when I previously said it only happens in 1% of cases. The user in question was unfortunately resizing their Cassandra cluster before the upgrade to 1.6.0. This caused the underlying stability issues necessary for the bug to occur.

## The bug

```go
var err error
if request.IsNewBranch {
    h.sortAncestors(branchInfo.Ancestors)
    treeInfoDataBlob, err := serialization.HistoryTreeInfoToBlob(&persistencespb.HistoryTreeInfo{
        BranchInfo: branchInfo,
        ForkTime:   timestamp.TimeNowPtrUtc(),
        Info:       request.Info,
    })

    if err != nil {
        return convertCommonErrors("AppendHistoryNodes", err)
    }

    batch := h.session.NewBatch(gocql.LoggedBatch)
    batch.Query(v2templateInsertTree,
        branchInfo.TreeId, branchInfo.BranchId, treeInfoDataBlob.Data, treeInfoDataBlob.EncodingType.String())
    batch.Query(v2templateUpsertData,
        branchInfo.TreeId, branchInfo.BranchId, request.NodeID, request.TransactionID, request.Events.Data, request.Events.EncodingType.String())
    err = h.session.ExecuteBatch(batch)
} else {
    query := h.session.Query(v2templateUpsertData,
        branchInfo.TreeId, branchInfo.BranchId, request.NodeID, request.TransactionID, request.Events.Data, request.Events.EncodingType.String())
    err = query.Exec()
}

if err != nil {
    return convertCommonErrors("AppendHistoryNodes", err)
}
return nil
```

*Buggy code*

Above is the original code. As you can see, `err` is declared as a proper `var` within the top level scope:

```go
var err error
if request.IsNewBranch {
    h.sortAncestors(branchInfo.Ancestors)
    treeInfoDataBlob, err := serialization.HistoryTreeInfoToBlob(&persistencespb.HistoryTreeInfo{
        BranchInfo: branchInfo,
        ForkTime:   timestamp.TimeNowPtrUtc(),
        Info:       request.Info,
    })
  ...
```

The intention of the author was that the subsequent call to `HistoryTreeInfoBlob` would set the previously defined `err` to the result of the operation err response. But in Golang, the `:=` operator is scoped. This means that because the call to 'HistoryTreeInfoBlob' is within the `if` branch, a new variable `err` will be created. If the call succeeds, the code proceeds to `ExecuteBatch`:

```go
    batch := h.session.NewBatch(gocql.LoggedBatch)
    batch.Query(v2templateInsertTree,
        branchInfo.TreeId, branchInfo.BranchId, treeInfoDataBlob.Data, treeInfoDataBlob.EncodingType.String())
    batch.Query(v2templateUpsertData,
        branchInfo.TreeId, branchInfo.BranchId, request.NodeID, request.TransactionID, request.Events.Data, request.Events.EncodingType.String())
    err = h.session.ExecuteBatch(batch)
```

If this call succeeds, all is well. But if it fails disaster strikes. The error will be assigned to the scoped version of `err`. The top level scope `err` will therefore not hold any value and the error handler guarding the return will not evaluate:

```go
if err != nil {
    return convertCommonErrors("AppendHistoryNodes", err)
}
return nil
```

The result of missing the `err` check is a positive response being returned to the caller even though the call to `AppendHistoryNodes` failed. To fix the bug we removed the nesting and no longer set the error within a nested scope. Here is the bug free version:

```go
if !request.IsNewBranch {
    query := h.session.Query(v2templateUpsertData,
        branchInfo.TreeId, branchInfo.BranchId, request.NodeID, request.TransactionID, request.Events.Data, request.Events.EncodingType.String())
    if err := query.Exec(); err != nil {
        return convertCommonErrors("AppendHistoryNodes", err)
    }
    return nil
}

// request.IsNewBranch == true

h.sortAncestors(branchInfo.Ancestors)
treeInfoDataBlob, err := serialization.HistoryTreeInfoToBlob(&persistencespb.HistoryTreeInfo{
    BranchInfo: branchInfo,
    ForkTime:   timestamp.TimeNowPtrUtc(),
    Info:       request.Info,
})
if err != nil {
    return convertCommonErrors("AppendHistoryNodes", err)
}

batch := h.session.NewBatch(gocql.LoggedBatch)
batch.Query(v2templateInsertTree,
    branchInfo.TreeId, branchInfo.BranchId, treeInfoDataBlob.Data, treeInfoDataBlob.EncodingType.String())
batch.Query(v2templateUpsertData,
    branchInfo.TreeId, branchInfo.BranchId, request.NodeID, request.TransactionID, request.Events.Data, request.Events.EncodingType.String())
if err = h.session.ExecuteBatch(batch); err != nil {
    return convertCommonErrors("AppendHistoryNodes", err)
}
return nil
```

*Bug free code*

As bad as it sounds it's a surprisingly difficult bug to reproduce. Cassandra has to succeed with one operation and fail with another within a very short window of time. Under normal operation this is very rare. Regardless, the bug can and did happen.

In general, we learned a big lesson about Golang shadowing. We know our Golang at Temporal, but many were still surprised about how shadowing works in all cases. Alex (our first engineer), confused all of this with the following example.

__What do you think it prints?__

```go
func main() {
    var err error
    defer func() {
        fmt.Println(err)
    }()
    if true {
        str, err := test()
        fmt.Println(str, err)
    }
}
func test() (string, error){
    return "str", errors.New("error")
}
```

__Output__

```go
str error
error
```

__How about now?__

```go
    func main() {
    var err error
    defer func() {
        fmt.Println(err)
    }()
    if true {
        str, err := test()
        fmt.Println(str, err)
    }
}
func test() (string, error){
    return "str", errors.New("error")
}
```

__Output__

```go
str error
<nil>
```

As you can see, in the right circumstances shadowing can be a complete nightmare.

## Conclusion

As with most software failures, humans (us) were the weakest point in the design. That being said, Golang came in a not so distant second. This incident has only strengthened our internal processes and culture and we plan to actionably do better moving forward.

As always, feel free to reach out with questions, comments or critical feedback via <a href="mailto:community@temporal.io">email</a>, [Slack](https://temporalio.slack.com) or our [community forum](https://community.temporal.io/).


I'm Ryland from Temporal - an MIT OSS platform for building highly reliable distributed applications. We pride ourselves on the reliability and consistency Temporal provides to our users mission critical applications. 

How Go shadowing and bad choices caused our first data loss bug in years

Head of Product

Ryland Goldstein

ryland goldstein

In a previous post, about the making of our SDK tutorials, I talked a bit about a tool called Snipsync. In essence it merges source code snippets directly into our documentation, and we have been steadily converting our code examples to use it.

Thus far, in the context of our current documentation ecosystem it has been quite a wonderful little tool. And because it has been working well for us, I wanted share a bit more about how it works with all of you. This short video provides a decent overview of the functionality and how to use it:

<a href="https://youtu.be/_qEFcOYYjqU" class="embedly-card" data-card-width="100%" data-card-controls="0">Embedded content: https://youtu.be/_qEFcOYYjqU</a>

There are some considerations to using Snipsync however. The source code snippets need to make sense within the context of the documentation. Comments should be carefully selected, and additional variables and function calls within the snippet should be minimized. Essentially, a little extra effort is usually needed to align the code snippet and the documentation to work together. But even so, over the long term, it should help alleviate the painful process of scouring documentation to make sure code examples are up-to-date.

In a previous post, about the making of our SDK tutorials, I talked a bit about a tool called Snipsync. In essence it merges source code snippets directly into our documentation, and we have been steadily converting our code examples to use it.

How we keep our code examples fresh

SDE Technical Writer

Cully Wakelin

cully wakelin

__Latest Release at Time of Writing: V1.7.0__

Hey there everyone, it's been a while since the last update. Things have been __very__ busy at Temporal over the last few months and while I can't share everything yet, there are some nice updates I can share.

## Cassandra persistence issue

Until 1.6.0 there was a bug that only affects users running with Cassandra persistence. We highly recommend that all Temporal-Cassandra users __immediately upgrade__ to 1.6.0 (or the updated version of your current major/minor patch) and refrain from making changes to their Cassandra cluster until they’ve upgraded. For more details read:

[https://community.temporal.io/t/important-announcement-for-temporal-cassandra-users/1540](https://community.temporal.io/t/important-announcement-for-temporal-cassandra-users/1540)

We plan to provide a more detailed post-mortem in coming weeks. TL;DR; Golang variable shadowing is no bueno.

## Security

At a product level, security has been a major focus over the last few months. Security features have been requested for some time, but they were always hard to prioritize since Uber runs Cadence in a private network. As we started to collect feedback about the most desired Temporal features in mid 2020, it immediately became clear that security needed to be a priority. For context, when we started our security efforts in October, authentication and authorization were the most requested features by almost double the next highest request. We also knew that AuthN/AuthZ were baseline requirements for our cloud solution.

Throughout November and December of 2020, we made immense progress securing Temporal. One of our major goals with both authentication and authorization was to be very un-opinionated and reinvent as few wheels as possible. For these reasons, we decided to design authentication so it could support any OIDC compliant identity provider. In November we launched authentication beta for open source. We really appreciate everyone who helped validate and drive our design.

In parallel to the authentication work, we were also working on authorization. We wanted a killer authorization experience, but also needed to meticulously review existing system APIs to ensure access control was enforced everywhere it should be. The team really worked hard to get this feature out securely and quickly. This hard work paid off and we released authorization into open source by mid December.

Outside of explicit features, we also have quite a few other security related processes going behind the scenes:

- In August we started our SOC2 process. The team really came together over the last few months to change our processes and systems to meet compliance standards. We are now nearing the end of our initial SOC2 process which is a huge milestone for the company and technology. Cully specifically deserves a shoutout as he stepped in to lead the compliance effort without any background in compliance or in-house expertise to rely on.
- As part of the SOC2 process, we were also required to have our product (both cloud and open source) penetration tested. This penetration test took place at the beginning of January. Only a handful of non-critical issues were found, and __0__ issues were found in the core service. We have since fixed all issues that were discovered.
- We take security very seriously here at Temporal and part of that is knowing when we're out of our depth. To ensure we are doing absolutely everything possible to provide a secure Temporal experience, we have brought on a renowned security expert, [Travis McPeak](https://www.linkedin.com/in/travismcpeak/), to advise us part-time. He has been an invaluable asset in validating our ideas and guiding us towards secure solutions.

Temporal went from a system with essentially no access control mechanism, to a system which can locked down and secured with minimal effort. We're happy with the progress made, and security will continue to be a major focus moving forward.

## Cloud

### Why cloud?

One of the most common questions we get is:

__"What is your revenue model?"__

followed by:
__
"Why not support/licensing/on-prem?"__

These are valid questions, so I'll briefly explain why cloud is the right fit for us.

Our number one goal as a company is to __improve the lives of as many developers as possible__. We're definitely biased, but we believe Temporal fundamentally improves developers lives in a way that most technology products don't. Because of this, we believe the number one way to reach our goal is to get Temporal into the hands of as many developers as possible. So now the question becomes:

> How do we get Temporal into as many developers hands as possible?

The answer is complex, but at the highest level it really boils down to making Temporal more accessible. When we look at the aspects of Temporal which are the least accessible today, it's clear that running Temporal itself is the biggest barrier of entry. While there are numerous motivations for our choice to be a cloud company, this is the most important. We believe the cloud is the optimal path for getting Temporal into the hands of more developers, therefore improving their lives.

We are also a business and from a business point of view, the cloud just makes common sense. We pride ourselves on providing high quality "no strings attached" support and becoming a services company would have changed our incentives making that much harder. On-prem is another common revenue path, but we know that it would mean spending far more time debugging customer infrastructure, and far less time improving Temporal. Finally there is the licensing route, but we really like MIT and strongly believe that creating an enterprise version would be unhealthy for our open source community. Even if you don't agree with this rationale, I hope it at least provides some insight into thought process.

### The cloud journey

For a long time, a hosted version of Temporal was merely an idea. It was an idea that we really liked but there had always been more pressing priorities. For the first half of 2020, our time and energy was consumed by producing a high quality production release. Once we released, our focus shifted to our funding announcement and subsequent exit from stealth. So by the time we announced our funding, serious demand had grown for a cloud offering. It became clear that the cloud was no longer a hypothetical "nice to have", but instead a real blocker for Temporal adoption.

Towards the end of October, we began seriously ramping up our cloud efforts. In addition to our dedicated cloud team, our other teams were also making changes to the core service necessary to provide a secure and reliable hosted experience. Security was the largest of these changes, but coming in as a close second was the removal of Kafka from both the visibility and cross-DC code paths. Removing Kafka was a core requirement for our hosted service, but it was also a huge win for Temporal. Removing Kafka means one less dependency is required to run the system. In general the team did an amazing job delivering all the features blocking our cloud effort.

With all of the cloud blockers out of the way, we approached a handful of initial design partners who we felt would be a great fit. We started working with our first partner throughout December, and were able to onboard them shortly before Christmas. Getting our first customer is a monumental step, especially considering they are a $20B+ company. After onboarding our first design partner, we continued to gain momentum. Throughout the month of January we onboarded 3 more customers onto the cloud. We are continuing to onboard design partners, and plan to have O(8) customers by the end of the February. Considering the sizable backlog of interested customers, we would love to onboard users faster. But we pride ourselves on providing an incredibly reliable and consistent experience, and we don't want to take any risks by overextending ourselves. Eventually there will be enough Temporal Cloud for everyone.

If you are interested in Temporal Cloud, feel free to reach out (ryland@temporal.io, maxim@temporal.io) so we can get your name on the waiting list. If you are interested in helping us, we're hiring.

## SDKs

Another area of focus over the last few months has been our language SDKs. Historically, Temporal only officially supported Java and Golang as official language SDKs. There are [community contributed SDKs for Ruby, Python and a few others](https://docs.temporal.io/application-development). While we really appreciate these contributions, we can't provide any guarantees for them until they are integrated in officially. We know that supported languages is one of the most important factors for choosing a new technology. This is why we've put serious effort into expanding our official language support over the last few months.

### PHP SDK
Six months ago we were approached by a group of well respected open source developers in the PHP community - SpiralScout. SpiralScout are the masterminds behind the [Roadrunner](https://github.com/roadrunner-server/roadrunner) framework for PHP, so were super excited that they wanted to contribute an official language SDK. They've now been hard at work for almost 6 months and are nearing completion. Very soon, PHP will be the 3rd officially supported language for Temporal.

### TypeScript SDK
PHP isn't the only language on its way to Temporal. Work is also underway on a TypeScript SDK for Temporal. This SDK also went through the [standard proposal process](https://github.com/temporalio/proposals/pull/15), and we really appreciate everyone who left feedback and helped guide the design. As you know, we don't do timelines here at Temporal. That being said, I can guarantee that we want TypeScript support as much as you do.

## Docs
Instead of poorly explaining our progress with docs, I'll defer to resident expert and information architect Cully:

> There have been many contributions to [Temporal's documentation](https://docs.temporal.io/) over the last couple of months. Highlights include Server information and SDK guides. In regards to the Server, while we have refreshed some of the existing information, we have also published new information on [security](https://docs.temporal.io/server/security), [versions and dependencies](https://docs.temporal.io/clusters), [Elasticsearch setup](https://docs.temporal.io/clusters/how-to-integrate-elasticsearch-into-a-temporal-cluster), and [Server options](https://docs.temporal.io/server/options). In regards to SDKs, thanks to [Anton Titov](https://github.com/wolfy-j), there is now a brand new set of [PHP SDK guides](https://docs.temporal.io/php/introduction) for PHP developers. The Temporal Product Team appreciates all of the contributions the community has made and we look forward to bringing you even more content over the next several months.

## Community

### Slack and Discord

For those who remember, we ran a survey a while back about Discord and Slack. Some users later reached out and asked about the purpose of the survey and if we had any immediate plans to switch. To directly answer, there are no immediate plans to switch. With that in mind, our medium/long term plan is to move away from Slack. There's a lot of reasons we're not fans of Slack for community management, most of which are covered in my previous post. The biggest blocker stopping us from moving is lack of threads. We feel threads are an important mechanism for asynchronous community communication. Unfortunately Discord has no official plans for thread support and definitely no timeline (which we can respect). We will continue to evaluate the situation and will keep the community updated as things progress.

I've also included the results from the two primary questions in the survey. The first (image below): "Would you be open to trying Discord..." was only presented to users who said they had never tried Discord.

![slack-and-discord-1](//images.ctfassets.net/0uuz8ydxyd9p/3V9LVygrOHP7Br4LXw7jYU/3b898cbbc13c8e3fda908448db9fd3b1/slack-and-discord-1.png)

While the second (image below), was conversely only shown to users who said they had tried Discord. The results of both questions indicate that the community is very willing to consider Discord in the future.

![slack-and-discord-2](//images.ctfassets.net/0uuz8ydxyd9p/5FCJYgYG8z3V8xoMAT0XmT/c0d4b12955beaea655ee2c9c7178766f/slack-and-discord-2.png)

### Open office hours and other content

You may have noticed that there hasn't been an open office hours in a while and that other auxiliary content has also decreased in volume. The honest truth is that we are a very small team and the amount of workload has increased much faster than our headcount has. For the most part, this is an amazing sign for the technology and the business.

While all of this is true, we really love community focused events like the open office hours and are dedicated to bringing them back. The most surefire way for us to accomplish this is by hiring more talented and intelligent people like all of you who are reading this. This is a perfect segue to the next section which talks about our current hiring and open roles.

## Work with us (seriously we need amazing people like all of you)!

I want to start with a quote directly from Karl - our Head of People. Karl is really exceptional and is an amazing point of contact if you're ever curious what it would be like to work at Temporal:

Hiring is a top priority for Temporal and we're focusing on expanding our teams in all Engineering and Product groups which include Infra, Cloud, SDK, Server/OSS, the Temporal Web, Documentation, Developer Relations, Solutions Architecture, and Developer Success. We have multiple openings in the United States and are expanding our recruiting team's capacity three-fold as we add employees throughout 2021 and beyond.

Feel free to reach out to Karl at karl@temporal.io

On a personal note, I have to say that Temporal is the best company I could have ever imagined working at. Our technology is revolutionary, the founders are unmatched, the team is both the most intelligent and most kind I've ever worked with. This is all on top of the fact that our business model is already validated. Most importantly, working at Temporal means spending your days helping other developers achieve their dreams and goals. That alone should convince you to join.

## Temporal jobs

While we are [actively hiring for all roles](https://temporal.io/careers) at Temporal, we also recognize that developers want opportunities to work with Temporal and that companies want to hire the kind of developer that understands how to apply reliable distributed systems to solve business problems. Much as we would like you to work with us, it's also in our interest to help our users find each other! To that end, we are also listing external Temporal Developer Jobs as a free service to our community. Our first posting is already up - Nylas is hiring Workflows engineers in Toronto and Vancouver!

## Conclusion

Hopefully this longer form update answers the most hot topic questions you all have. We're incredibly excited with the progress we've made and what the future will bring. We couldn't have done it without everyone in the community supporting us and pushing us forward.

As always, feel free to reach out with questions, comments or critical feedback via <a href="mailto:community@temporal.io">email</a>, [Slack](https://temporalio.slack.com) or our [community forum](https://community.temporal.io/).


norbert-kowalczyk-Do6Uql09Lmk-unsplash

Hey there everyone, it's been a while since the last update. Things have been very busy at Temporal over the last few months and while I can't share everything yet, there are some nice updates I can share.

Temporal Transparency Update #10

__Latest Release at Time of Writing:__ V1.5.1

I recently gave a talk at the CodeMesh conference, and I spent half of it reflecting on the seemingly boring topic of dealing with failures. The talk was primarily based on my experience building and helping others build cloud services with the Orleans framework. I chose this topic, because I believe dealing with failures is the most important aspect of any system. Oftentimes, it is what stands between a product that runs as expected and one that keeps producing surprises and causing investigations. When done right, handling of failures is what differentiates a professional from an amateur.

The talk covered three approaches that I've seen and applied the most myself:

- Request-Reply (a.k.a RPC)
- Using persistent queues
- Workflows

## 1. Request-Reply

In my opinion, Request-Reply ([a.k.a. RPC -- Remote Procedure Call](https://en.wikipedia.org/wiki/Remote_procedure_call)), is the most natural way of handling failures. The client makes a request to the server and waits for a response (up to a timeout) and in most cases learns about a request processing failure immediately. This is how HTTP works, for example.

> Note that by client and server I mean simply two sides of the call. They can be real client and server processes or merely two tiny objects communicating with each other within a distributed system.

Simplicity of RPC is good for the server.

> "I try to do what the request asked me to. If there's any failure downstream, I return it to the client. The client knows best what to do, to retry or not, how many times, with a backoff or not. My logic can stay simple."

In our world of overly complicated systems, the value of simplicity is difficult to overstate. However, in this case the complexity burden gets pushed to the client. This puts the remote client at a disadvantage here. It has to operate based on the limited error information it received back. Sometimes it’s just a communication error or a timeout. These are a few of the many possible real life cases:

- An error may not be clear about whether the requested operation actually failed. It might have succeeded, and the error happened while trying to communicate success. This forces the client to either check for the status of the operations or retry anyway, assuming retrying the operation can be done safely, i.e. it is idempotent.
- The system may be temporarily unavailable, actually being down or network partitioned from the client. For mobile applications that's rather expected.
- Partial failures are hard to deal with. When we need to update multiple external systems at once, there is almost never a way to do that in an all-or-nothing manner, i.e. atomically. So, we have to handle retries and rollbacks, side effects, and all the inevitable complexity.

The last point I illustrated with the following picture:

![dealing-with-failure](//images.ctfassets.net/0uuz8ydxyd9p/4o6euyTyujLP38MK9peZOt/a9df19172088b47cdf9dc3cea312cc64/dealing-with-failure.png)

In this example, the client (square blue thing) makes a request to the server (round green thing). The server does not have the information locally to satisfy the client request and therefore needs to call two external services, blue and purple.

If either of those two sub-calls fail, the server returns an error to the client. If the client were to retry the request, there would need to be a mechanism in the server that prevents duplicate calls to the external services.Idempotency is one method of addressing this issue. If the client decides to give up, there needs to be a way to revert any changes made as part of processing the request before the failure (in our example - of the call to the service A).

A canonical example is money transfer from an account in one bank to an account in a different bank. However, there are many other scenarios with conceptually indentical requirements. In many cloud orchestration cases we need to allocate a resource (such as a virtual machine), and then perform a number of operations with it before returning it in a ready state to the client. If any operation fails, we don’t want to leave the VM running. Nor do we want to keep allocating new VMs for the same request.

To summarize the pros and cons of the RPC approach.

Pros:

- Simplicity
- Obvious correlation between a request and a failure

Cons:

- Retries are client's responsibility and are difficult to do for a remote client
- Partial failures are difficult to handle

## 2. Persistent Queues

Putting a persistent queue between the client and server solves a number of problems. The client just needs to successfully send a request to the queue to ensure that it will eventually be processed.

![dealing-with-failure-1](//images.ctfassets.net/0uuz8ydxyd9p/41w1tIy8O5XxHWZ24FZyY7/7c368a3198b94ffe4d317e7061c5f117/dealing-with-failure-1.png)

Assuming the server only deletes a request from the queue after it is successfully processed, we get a simple retry mechanism. Due to the queue, even if the server crashes and restarts in between the attempts, it will keep trying to process the request again and again. The fact that the client (producer) is completely decoupled from the server (consumer), means the client can enqueue requests even if the server is down. This is the main reason why the publisher-subscriber architecture is so popular. Separation of subsystems in space and time is a nice property.

![dealing-with-failure-2](//images.ctfassets.net/0uuz8ydxyd9p/0Uhb4S2C2BozU5gXXPrkB/d93f7b298bade3077ca97b323cbcc11c/dealing-with-failure-2.jpeg)

*A simulated illustration of Chang'e-5 probe's orbiter-returner's separation from the ascender on the moon orbit, December 6, 2020. /CNSA*

For streaming one-way events, queues are great. But how can the client get a response in a queue based architecture? There's no good answer to this question that I'm aware of. Responses need to be delivered (somehow) back to the client, usually over another queue. Then the client needs a way to correlate requests and responses, typically done via correlation IDs. There also needs to be timeout mechanisms for dealing with requests that never received a response.

Retries are simpler with queues compared to the RPC case. They are pretty much automatic, as long as the request stays in the queue. Calls to external services still need to be idempotent. However, we can't retry forever and have to deal with requests that keep failing to process. Either because they clog the queue (if the queue is ordered), consume too many resources or cause excessive load on the external services. The popular approach is to treat such requests as "poison messages", by moving them out of the queue to a different location ("dead letter" queue) for special handling.

Pros:

- Separation of systems in space and time
- Automatic retries
- Simple when no responses are expected

Cons:

- Additional dependency of the queueing technology
- Extra work to correlate responses
- Queues may clog
- Special handling of "poison messages"

## 3. Workflows

Similar to queues, workflows take the burden of ensuring successful execution of requests off the client's shoulders. But instead of writing them into a shared queue, requests are persisted as part of an independent workflow document. That document makes processing requests stateful:

- Tracking which steps of processing succeeded
- Tracking which steps of processing failed
- Remembering how many retries have been made, etc
- Workflow

Workflows have other important properties and use cases. They are a great way to implement long-running business processes, incorporate human operations and react to events. From the failure handling perspective, the most important aspect of workflows is the ability to be more intelligent when handling partial failures. Instead of being oblivious about what happened in the past, a workflow can keep a log of all relevant information and make informed decisions about what to retry and when.

Workflows can be individually addressable, which makes them easier to scale compared with shared queues. It also allows for targeted inspection and even on-the-fly modification of their state, if needed.

At the same time, workflows "inherit" most of the challenges of queues. Responses still need to be correlated with requests, although the individual addressability of workflows makes it easier for the client to query results. "Poison messages" are also still possible. They don't clog the queue anymore, but still require special handling.

Pros:

- Partial separation of systems in time and space
- Robust handling of partial failures
- Support for long-running operation
- Retries are "automatic"

Cons:

- Additional dependency on a workflow system or complexity of in-house implementation
- Extra work to correlate responses
- Special handling of "poison messages"

## Conclusion

It's a cliché that in our business there's no free lunch, only tradeoffs. As unoriginal as they might sound, many clichés are true. Dealing with failures is an area of important tradeoffs. There's obviously no single pattern that fits all scenarios. In fact, many systems leverage all three patterns I described.

For simpler requests that need a prompt response and aren’t involved with complex multi-step processing, Request-Reply is often the right approach. One-way messages, events, data streams are clear candidates for Queues. Workflows are a good fit for reliable execution of relatively complex requests that either require multi-step processing or can leak resources if failures aren't properly handled.

omar-flores-lQT bOWtysE-unsplash

I recently gave a talk at the CodeMesh conference, and I spent half of it reflecting on the seemingly boring topic of dealing with failures. 

Dealing with failure

Sergey Bykov

sergey bykov

There is nothing more frustrating than a cryptic error message.

When something goes wrong, chances are you already knew it - all a cryptic error message does is confirm your suspicion and tell you that you're gonna need to read through a bunch of code to figure out your error.

As framework and library designers, we can and should do better for our users by carefully following a few design principles.

## Why We Should Care

Who among us hasn't thrown up their hands at an unhelpful `Error: NullPointerException or undefined is not a function`? Of course, this is a [straw man](https://en.wikipedia.org/wiki/Straw_man) - unanticipated errors always leak internals. We should do our best to handle these errors as much as possible!

The frustrating part comes from errors that were anticipated but obviously not given any thought:

![error-message-design-1](//images.ctfassets.net/0uuz8ydxyd9p/Uh5J6MCRURtfB1cevz49x/88dfa9b8f3c0b83a3ada5ad7dfa61575/error-message-design-1.png)

This is particularly aggravating because it compounds an already unhappy situation with an even worse debugging and bug reporting experience.

Thoughtful error message design is one of the less glamorous, but most important, parts of [Developer Exception Engineering](https://www.swyx.io/developer-exception). Most framework developers instinctively write error messages that make sense to them, but a few minutes of extra effort can save DAYS of debugging for everyone else (including maintainers!).

We must acknowledge two things:

1. there is immense information asymmetry between the framework developer and the framework user (developers know the codebase far better than users)
2. error messages are part of your developer experience too

You might be familiar with the old adage "If you fail to plan, you plan to fail." Here, we have a slight twist: __If you fail to plan to fail, you are planning to fail your users when they need you most__. Touting great developer experience with lousy error message design is like selling a beautiful car with seatbelts that don't work when there's a crash.

## Error Message Style Guide

At Temporal, our current style guide reflects our engineering roots in [Uber](https://github.com/uber-go/guide/blob/master/style.md#error-types) and [Golang](https://go.dev/blog/go1.13-errors) with some tactical recommendations:

- Use `errors.New` for simple messages and `fmt.Errorf` for formatted messages. Make sure you're using the standard `errors` go package.
- Error messages start with lowercase and no punctuation at the end: `unable to open file` instead of `Unable to open file..`
- Error messages should be actionable, e.g. `no such directory` instead of `loading service config`.
- Use `unable to...` instead of `failed to...`, `can't...`, or `could not....`
- Messages should reflect the logical action, not merely a function name: `load service config` instead of `LoadServiceConfig`. This message should add actionable, meaningful context to the existing error message.

That said, as we evolve our developer experience philosophy, we are looking toward higher level design principles that help guide successful outcomes.

## Error Message Design Principles

We are still in the early stages of pinning down our beliefs in what makes for great error messages, but here are some candidate principles running through my mind:

- __Errors should be written for USERS, not maintainers__: Your error should make sense from a user point of view. For example, you should cite user-supplied configs and values familiar to users, not the internal variable and function names familiar to maintainers.
- __Errors should suggest WHY they happened__, or even better - HOW to fix them: As a maintainer, you probably have a high amount of context as to the possible causes of an error - take the time to write down some hints for users. If there's a high probability you know what's wrong, the error message itself can suggest how to fix it. But even without that certainty, think about what the user is likely to need to log out to debug the error, and offer it in the error message itself. Where relevant, __Errors should display what was expected vs what was received__.
- __Errors should be searchable__: Your users are most likely going to paste your errors into the search bars of your issue tracker, docs site, or Google. Consider the "SEO" of your errors. Sometimes offering a unique, human readable ID for each error can help narrow things down very quickly!
- __Errors should be logged__: Errors that happen far more often than they ought to are a signal of a deeper issue. If a user code or configuration error happens too often, it isn't the user's fault - you have bad API design. If inside a terminal environment, you should offer some text formatting to make your errors visually distinct from normal log messages.
- Last but not least, __Errors should not be too verbose__: You might be tempted to dump every piece of information under the sun, but this can be very annoying when scrolling through reams of errors. Most users don't read entire errors - make your words count, and link out to docs for fuller context. Errors are not docs.

Ultimately, the same high level principle applies to error messages as they do for anything involving human computer interaction: [Don't Make Me Think](https://www.amazon.ca/Dont-Make-Think-Revisited-Usability-dp-0321965515/dp/0321965515/ref=dp_ob_image_bk)!


shubham-dhage-C8YfO3YStVo-unsplash

There is nothing more frustrating than a cryptic error message.

Write Errors That Don't Make Me Think

![the-making-of-new-sdks-1](//images.ctfassets.net/0uuz8ydxyd9p/5o04wZEcx3xyZNz1fQ0dfD/ee50a80e4058acc486bcc986d6ff7c0c/the-making-of-new-sdks-1.png)

In early September it was clear that V1 of Temporal would be ready by the end of that month and we would be announcing our funding shortly thereafter. From a documentation perspective, we felt that it was important to coordinate changes in a way that would support the announcement. As with any product launch, we were hoping to create some buzz and see a surge in new users. Considering that documentation is one of the most important aspects of new user adoption, we had our work cut out for us.

## Uphill challenges

![the-making-of-new-sdks-2](//images.ctfassets.net/0uuz8ydxyd9p/4yTFQ6Vo86tgVyxH6zh1Ra/8872bb80daf33b4a1694b29319c95f23/the-making-of-new-sdks-2.png)

In terms of our documentation, there were at least three challenges we were facing. Two of those challenges stemmed directly from the fact that our docs started as a fork of the docs from Temporal's predecessor.

The first challenge is that the information we inherited lagged behind in fully describing Temporal's capability and feature set. One of the reasons for this is that documentation is typically offered a secondary level of prioritization. While Temporal now prioritizes documentation, this was not always true from where Temporal originated as Cadence.

The second challenge was that there have been many core changes to the system, terminology, and SDKs in the time since Temporal forked from its predecessor. Back in early September, many of these changes had yet to be propagated throughout the docs as well. So, not only was there missing information but some of the existing information was just plain incorrect.

The final and biggest challenge of documenting Temporal is that it is fundamentally new. Temporal presents a different approach to application development. Users are faced with a set of familiar terms and concepts but must comprehend them in an entirely new context and landscape.

## Picking a path

![the-making-of-new-sdks-3](//images.ctfassets.net/0uuz8ydxyd9p/2jG35nsZ9VqjP6WuztKtdx/550608392176d175bfe418342acd7d8f/the-making-of-new-sdks-3.png)

At a high level, there are two parts to Temporal: the backend and a client-side SDK. Configuring, deploying, and operating the Temporal backend for a live environment is no small task. On the other hand, it is really easy to [get Temporal running on your local machine](https://docs.temporal.io/clusters/quick-install) in a Docker container. In fact, you can do it with just two terminal commands.

The Docker route definitely simplifies running the backend, which means the majority of friction for new users comes from our SDKs ([Go](https://github.com/temporalio/sdk-go), [Java](https://github.com/temporalio/sdk-java)). While an SDK is meant to abstract the complexities of interacting with the server API, Temporal flips a lot of the preconceived notions of modern application development on their head. The SDK docs needed to do more than just provide example usage. They also needed to show the "why" to enable the user to grasp the concepts that Temporal is promoting. So we went about scoping something that we could realistically accomplish within that time frame and still be relatively effective.

We decided that the best thing for us to focus on was a great new user experience. We wanted something that would enable a developer to start using the product right away but also leave them with an understanding of the value Temporal provides. We wanted to cultivate that "aha!" moment.

We started as most might, by trying to envision what the ideal new user experience would look like. We then identified as many of the steps it would take to get there as possible. Looking back, I would contend that we managed to lock onto three ideas that we thought would get us closer to the ideal experience. The hope was that once these three ideas were combined they would result in a set of effective SDK tutorials for new users.

## Snipsync

![the-making-of-new-sdks-4](//images.ctfassets.net/0uuz8ydxyd9p/5NN7p075VyLsOuELG1XS4K/694144c1a4caad7db82528a583f45adb/the-making-of-new-sdks-4.png)

It was around this time (early September), that I was testing out a Node.js tool I had built to improve the experience of creating and maintaining documentation. It downloads Github repos, scrapes code snippets that exist between specific comment wrappers, and writes the snippets to their corresponding comment wrappers in Markdown files.

Source code

```
// @@@SNIPSTART unique-name-of-snippet
SomeFunc() {}
// @@@SNIPEND
```

Markdown file

```


```

We borrowed the idea from [Google's proprietary version](https://github.com/GoogleCloudPlatform/golang-samples/blob/main/secretmanager/get_secret.go#L17) they use for their [Google Cloud documentation](https://cloud.google.com/docs). The concept is fairly simple, and I am still surprised that there was no existing open-source solution. So we made one!

A tool that automates the synchronization of code with the docs from any given repository has several key benefits:

1. Code snippets in the documentation are guaranteed to work as they are continuously tested. This also means that they can be reliably copied and pasted into the user's editor.
2. We control exactly which lines of code are shown and can also target a specific branch or commit. This is a great safeguard against bugs which might be introduced to the main branch.
3. We never have to commit source code into the docs. The code is merged into the Markdown at build time. This ensures that the code is already reviewed and approved from the repo it resides in.

Snipsync does come with a few considerations:

1. Embedded code needs to have carefully reviewed comments, structure, and make sense within the context of the documentation. For example, if the code snippet is coming from a working repo it may include additional variables or function calls. These must be minimized and optimized out so they don’t cause unnecessary confusion.
2. In the same way that the code must be optimized for the docs, the docs must also be optimized for the code. In essence, the docs are being “driven” and "defined" by the underlying code. And if no one has coined the term yet, I think the credit for “code-driven documentation” should go to our Head of Product, Ryland Goldstein, as he pinged me one afternoon to share that epiphany with me.

![the-making-of-new-sdks-5](//images.ctfassets.net/0uuz8ydxyd9p/2450tNU2QBK0eXblhRHm0s/9b0cbcc9f8403567a9d17069264321c1/the-making-of-new-sdks-5.png)

We decided to embrace [Snipsync](https://github.com/temporalio/snipsync) as the challenges it introduced were minimal compared to the value.

## Template repos

![the-making-of-new-sdks-6](//images.ctfassets.net/0uuz8ydxyd9p/3QLdpi5h1qYs5n2RVpfRkh/977003ff2b370c3f3c8a8b50b5aca8fd/the-making-of-new-sdks-6.png)

We now had a way to synchronize code with our documentation. But from where will the code be synchronized? We know users will likely want to view the source file and relative file path of the code snippet for added context. They will also be likely to clone the repo and try to run the sample.

We actually already repositories of code samples for the [Go SDK](https://github.com/temporalio/samples-go) and [Java SDK](https://github.com/temporalio/samples-java). While we desired to see more samples, there were already quite a few of them in each repository. But, we discovered that shared sample repositories tend to have two issues that made them less ideal for synchronizing with docs.

1. While convenient, colocating multiple samples in a single repo is far less approachable compared with storing samples in self-contained repos.
2. In shared sample repositories, it is harder to retain the idiomatic nature of any language and mirror the functionality of any sample across different languages.

So for each of the sample applications we planned on using to drive our documentation, we created a corresponding template repo. These template repos can be easily copied, built, and run in a matter of minutes.

- [Go money transfer template repo](https://github.com/temporalio/money-transfer-project-template-go)
- [Go hello world template repo](https://github.com/temporalio/hello-world-project-template-go)
- [Java money transfer template repo](https://github.com/temporalio/money-transfer-project-template-java)
- [Java hello world template repo](https://github.com/temporalio/hello-world-project-template-java)

## The tutorial

![the-making-of-new-sdks-7](//images.ctfassets.net/0uuz8ydxyd9p/7y68ovrQb8oXWvm9YWmsCS/e3a5231a191349a7b2af382344341b23/the-making-of-new-sdks-7.png)

Since the goal of our documentation changes was to help with new user acquisition, we decided to aim for a "tutorial" style of documentation. The first iterations aimed to build upon and replace the existing SDK “quick start” pages that maintained the status quo and printed “Hello World!” to the console. As you may have guessed, this route wasn’t sufficient enough to show users the real value Temporal offers.

Once it became clear that a standard approach wasn't going to cut it, we brought in our co-founder and CEO, Maxim Fateev. We asked him to give us a demonstration that he typically uses to introduce engineers to Temporal for the first time. The scenario represents a money transfer from one bank account to another and during the demo Maxim demonstrates what happens if one of the steps in the transfer fails. The money transfer sample was a great way to introduce the values of Temporal. For if you understand the ramifications of losing money from a failed financial transaction, several values of Temporal become immediately apparent:

1. The state of running code is maintained even through hardware failures, server crashes, and network outages.
2. There is deep visibility into the state of code execution out of the box via the CLI or UI.
3. Function calls come with automatic and retries and configurable timeouts.
4. Bugs can be hot fixed in running code.

For someone that is new to Temporal the attraction doesn’t come from using the SDK to print “Hello World!”. Instead it comes from witnessing the inherent benefits that Temporal offers by running simulations using a pre-built application.

This is the direction that we decided to send new users. If a user can wrap their heads around the value that Temporal brings to their application right out of the box, then spending time and energy on application setup and learning the SDK becomes a non-blocker.

Check out these finished tutorials and see for yourself!

- [Go: Run your first Temporal application](https://learn.temporal.io/getting_started/go/first_program_in_go/)
- [Java: Run your first Temporal application](https://learn.temporal.io/getting_started/java/first_program_in_java/)

## Next steps

![the-making-of-new-sdks-8](//images.ctfassets.net/0uuz8ydxyd9p/4FRP8K8idnJlxuzvv2XsWW/2591afb28ff6939b64e0a6201b2bd0c5/the-making-of-new-sdks-8.png)

At Temporal we understand that our documentation plays a very important role in our users' experience. And to get our docs into a world class state we have lots of work in front of us. In the near future we will be looking at the end-to-end journey through our documentation and how we can provide the best experience for every user. To validate any direction we take, we will be engaging with the community to hear what you think and help us dial things in. Any user can [schedule a 15 minute feedback session](https://calendly.com/cully-temporal/schedule-feedback-session) with me directly! We will also be preparing for all the new and exciting features around our hosted cloud offering that will enable all developers to build invincible applications.

In early September it was clear that V1 of Temporal would be ready by the end of that month and we would be announcing our funding shortly thereafter.

The process of creating SDK tutorials for new users

From a software development standpoint, Temporal is in the unique position of acting as both a database and a service orchestrator. 

Deep dive into Temporal stress testing

Manu Srivastava

manu srivastava

Why Temporal is moving to Discourse for our knowledge DB needs.

## TL;DR; 

The growth of this awesome community has made it difficult for us to provide the desired level of support using our current tools. A solution is needed that will scale with our communities growth and we strongly believe that Slack is not this solution. To solve this, we have started a Discourse forum and will slowly migrate certain questions/discussions away from Slack. We believe that Discourse will allow us to maintain the same responsive and personal support that we offer today, but with far less overhead.

Check it out here at: [https://community.temporal.io/](https://community.temporal.io/)

## Rationale for Slack Migration and Discourse

A couple months ago we reached the message limit on Slack and our community started asking if we planned to pay. While paying would have been the quickest solution, Slack is so expensive that we had to consider alternatives. How expensive you ask? Slack charges a minimum of $7 per user each month but more realistically $12 per user each month. With our current community of ~500 we would be paying anywhere from $3500 - $6000 each month. As we're expecting at least ~10x growth over the next few years the problem becomes much bigger as $60k a month isn't feasible.

Initially we were searching for a cheaper Slack alternative but we quickly realized that if we were considering a migration, it made sense to evaluate alternatives holistically and not just based on the fact that Slack costs more. To keep things consistent we defined a number of metrics to evaluate alternatives by:

- Cost
- Searchability
- Indexed on the web
- Responsiveness (real-time vs high latency)
- Accessibility
- Moderation and community safety
- Onboarding overhead
- Thread support

While these factors did not provide a clear answer to what our new solution should be, it did suggest that Slack was not going to work for us long term. Before I explain why Slack isn't a good fit for us, I want to say that there are a ton of really great things about Slack:

- Responsive
- Thread support
- Searchable internally
- Widely adopted
- Great for teams
- API

On the other hand there are a ton of issues when using Slack for a community:

- Incredibly expensive
- Made for teams, not communities (this is an intentional choice by Slack)
- Only searchable if you pay
- Not indexed on the web
- Zero to little moderation tools
- Onboarding is tough
- Not the most accessible

## Starting Our Search

At first we searched for a 1:1 replacement for Slack, a Slack clone if you will. This proved difficult as there are few tools that met our criteria (Mattermost, Rocketchat) and were widely adopted enough for us to move forward with. In an effort to quantify our process, we began to analyze exactly what value Slack was providing us. Our conclusion was that Slack was really serving two purposes for our community:

1. Knowledge DB (think Wiki)
2. Realtime discussions, casual chat, scheduling, announcements etc

Looking at Slack in this context suggested that the best move might be to look for two tools to replace Slack instead of one. While adding more tools is not my favorite thing as a product manager, it's much better than having 1 tool which inadequately serves both needs. Splitting up the task also meant that we could prioritize our efforts. While replacing the realtime portion of Slack is something we will need long term, we were experiencing far more issues with the knowledge database so we decided to tackle that first.

To start our knowledge DB search we defined our core requirements for a tool. We knew that search was a top priority, it needed to be fast with no limits on history. It was also important that users could rely on their search engine and not just the tool itself to find answers to their problems. This requirement eliminated a lot of Slack-like tools as they are not indexed on the web. Robust moderation capabilities were also a must. We've seen how Slack can create real safety problems for large communities due to the lack of fine grained user permissions and moderation. Our last major requirement was that the tool would enable us to maintain the same level of support responsiveness as we have with Slack. This community is the basis of our business so choosing a new tool which would negatively affect our support experience wasn't an option.

After thorough research it seemed like a traditional forum was the only option which could meet all of our criteria. We evaluated a few solutions (Discourse, Flarum, Tribe) and eventually ended up going with Discourse because it's:

- Widely adopted
- Open source
- Based around communities
- A Ton of companies that we align with ideologically are using it (Netlify, Docker, Rust, Elastic, etc)
- Robust moderation tools
- Very accessible
- Publicly indexed on the web
- In-site search is awesome
- Accepted answers feature
- Fine grained notification controls
- Great onboarding process - can even sign in with Google/SSO
- Affordable

Before fully committing to Discourse we trialed the tool internally. The more we used the product and became familiar the more we loved it. Furthermore, the support team at Discourse is seriously outstanding and they answer our questions in literal minutes. I hope this sheds some light on the decision making process behind this migration.

## How Will This Transition Work?

While we're fairly confident that Discourse will be a great fit for our community and team, we're not going to finalize anything until we see it in action. For now, we plan to stay in an evaluation period so we can ensure that our new tool doesn't degrade the existing support experience. You will see us push to migrate some conversations to Discourse from Slack when we feel that they have gone out of scope. We're also considering adding a Slack plugin which can automatically migrate Slack threads to Discourse. The eventual goal is to field all support questions via Discourse and keep a Slack/Discord/etc as a medium for casual discussion and brainstorming. To be 100% clear, we will not be removing Slack in the immediate future although this may happen down the road.

Most importantly we want to hear your feedback/criticism/suggestions about this change or our support experience in general. If you have ideas that you think would improve the community or our support experience please do not hesitate to pm me on Slack or via email (<a href="mailto:ryland@temporal.io">ryland@temporal.io</a>).

Lastly, if anyone is interested in being a moderator on the community forums, let me know and we can discuss what that process would look like.

[community.temporal.io](http://community.temporal.io)

Why Temporal is moving to Discourse for our knowledge DB needs.