Instacart: Data Quality Checks at Scale with Temporal

Abstract

At Instacart, the constant evolution of our product can quickly invalidate assumptions about data ingestion, transformation, and consumption. Without robust data validation, quality issues often go unnoticed until it’s too late, impacting internal analytics and even exposing bad data to our customers through ML models and data products. Thus, data quality becomes paramount.

Historically, Instacart has grappled with a fragmented landscape of data quality tools, leading to inconsistent adoption and hindering the ability to measure data quality coverage across the company. Recognizing the need for a unified solution, we developed DDC (Declarative Data Checks) – a powerful, secure, and compliant data validation platform.

DDC allows users to define data quality checks using a user-friendly UI, with built-in AI features to help generate optimal check definitions and SQL queries. The platform is designed to be highly configurable, driven by simple YAML files, and can connect to any database technology.

Under the hood, DDC leverages the power of Temporal to execute user-defined checks on a schedule, ensuring scalability and reliability. When issues are detected, DDC seamlessly integrates with Datadog for real-time alerting, keeping our teams informed and empowered to act quickly.

In this talk, we’ll dive deep into the architecture of DDC and explore how Temporal has been instrumental in building this mission-critical system. We’ll share our journey, challenges, and the lessons learned along the way. Join us to discover how Instacart is revolutionizing data quality management and ensuring the integrity of our data-driven decisions.

About the Presenter

Anant Agarwal is a Software Engineer at Instacart where over the past 3+ years he has built a variety of platforms and internal frameworks to improve developer and data productivity. Currently, he works within the Data Infra group, focusing on creating systems and tooling that simplify data workflows for both technical and non-technical internal users. Before Instacart, Anant worked at VMware on their VMware Cloud on AWS and High Availability offerings.