AI Data Workflows and 4-6x Developer Velocity at Messari

Thanks to Temporal, we have 2-3x faster developer velocity with 2x fewer engineers to achieve the same outcomes.

Messari is the Bloomberg for crypto and provides market intelligence products that help investors navigate digital assets. They have two major business components: they sell a comprehensive suite of data visualization and asset discovery tools, and they deliver industry-leading research that provides deep market insights and helps customers optimize investment strategies.

Messari is ultimately a data business whose backend platform is critical to its competitive advantage and the team is always looking to optimize its data platform.

They recently turned to Temporal to help them improve the reliability of their data ingestion pipeline that collects massive amounts of data from various sources and then enriches it with AI. This process previously relied heavily on cron jobs. This design worked well, however these jobs were difficult to debug at scale because they needed more controls and observability. As Messari business expanded, a need for more scale, resilience, and observability became increasingly important..

As they looked to rethink this ingestion flow, they wanted to avoid cron jobs and queues, and they didn’t want to create a custom orchestration system to oversee and ensure the work was completed reliably.

Before Temporal, we had to code for dead letter queues, circuit breakers, etc., to ensure we were resilient to potential system failures. Now we eliminate these complexities

Improved Resilience and Observability with Temporal

Temporal gives Messari resilience and observability for their data ingestion pipeline. It also helps reduce code complexity because it simplifies the debug process and allows them to set retry policies for failed activities instead of having to add this logic.

For instance, with Temporal Workflows, the Messari team can now systematically track retries and if they take too long, they can debug the issue in Temporal and retry the workflow or activity again.

With Temporal, the headache of maintaining custom retry logic vanishes.

The data ingestion process also relies on third-party APIs (such as X) to collect and enhance data. With the task rate limiting capabilities of Temporal, they can better overcome the limits of these external calls and provide a more smooth and reliable experience without having to code for any unreliable interactions.

Messari evaluated Airflow as part of this process, but felt it lacked the necessary observability they needed to debug and they are a Golang shop, so the Temporal Go SDK was a great fit and allowed their development team to just build and not learn another DSL.

Scale with Temporal Cloud

The Messari team chose to use Temporal Cloud not only to offload the operations of this service, but because it also provided them with a simple and effective way to increase capacity on demand. Messari can now scale out their application and Temporal Cloud scales to meet this need and they only pay for the infrastructure they need at any given time .

More features, faster Messari measured an improvement of 2-3x faster delivery of features and has used half the engineering time to achieve the same development outcomes by using Temporal. They have also sunsetted their existing boilerplate retry logic, queues, and cron jobs and simplified their codebase.

Temporal also enables their new engineers to ramp up faster because workflows are easier to understand than the legacy code. Messari can use junior engineers to build sophisticated products with much less oversight because of the strong guarantees that Temporal gives them.

Before Temporal, we would have to run manual scripts to do ugly fixes to production to fix bugs or outages. We don’t do that anymore