| A conversation with Frank Hidalgo, CTO of StackAI, now part of Asana and Hien Phan, Temporal's VP of Product Marketing. |
|---|
TL;DR#
-
The application: StackAI turns files from Google Drive, SharePoint, and Dropbox into searchable knowledge for enterprise AI workflows.
-
Temporal's role: Temporal coordinates the download, extraction, chunking, and indexing stages, records their progress, shows engineers where work failed, and retries only the failed stage instead of starting over.
-
The result: Indexing errors fell from about 20% to below 0.1%, and customer complaints about indexing stopped. Along with changes to parsing, caching, and retry logic, Temporal helped cut parsing costs to roughly one-tenth of their previous level.
Connecting Google Drive to an AI workflow can look like a one-click feature. The files are already there. The model just needs to read them, right?
StackAI gives regulated enterprises a no-code way to adopt AI with governance and guardrails. Its knowledge bases combine information from SharePoint, Google Drive, and Dropbox so customers can search that information and use it in automated workflows.
Behind that click, StackAI has to download every file, extract its text and images, split the content into chunks, write those chunks to a vector database, and make them searchable. An API rate limit, failed retry, parser failure, or unexpected file can break the chain.
When Frank Hidalgo joined StackAI to lead indexing and RAG, that chain broke too often: about one file in five failed to index on the first attempt.
This conversation has been edited and condensed for clarity.
What happens after a customer connects a drive#
Hien: What actually happens after a customer connects a source such as Google Drive?
Frank: The drive might contain a thousand files in many formats: PDFs, images, Word documents, text files, and files of different sizes. The first time a customer connects it, we have to download every file, and that is already a point of failure: API rate limits, connection problems, or transient errors.
Next we extract the text and images, using a combination of third-party services and our own code. A service can fail, a file can contain something unexpected, or our parsing code can hit an edge case. Then we split the content into chunks and index it in a vector database, which is another point of failure. Every stage has to succeed for a file to become searchable.
| Temporal Cloud coordinates StackAI's Workers through download, extraction, chunking, and indexing, and retries a failed stage without rerunning completed work. Search queries the same vector database. |
|---|
Where the original pipeline struggled#
Hien: What was the pipeline like when you took responsibility for indexing and RAG?
Frank: I joined StackAI about 14 months before this interview to lead indexing and retrieval-augmented generation. The original pipeline was built on Celery, with Flower for visibility. It had served the company's early growth, but observability, error handling, and retries were limited, and the failure rate was around 20 percent.
The pipeline was I/O-heavy. Our Celery setup ran synchronously, so we used threads for parallelism, and we could run out of workers or memory. When a later step failed, a retry repeated work that had already succeeded, including paid API calls.
Moving one stage at a time#
Hien: How did you move the indexing pipeline to Temporal?
Frank: We did it sequentially on Temporal Cloud, so there was never a big-bang rewrite. Imagine that the knowledge-base pipeline has ten stages. We made stage one durable with Temporal, then stage two, and kept going until every stage ran on Temporal. From then on, anything that failed, I could have my eyes on it: the file, the stage, and the error message. And Temporal retries only the step that failed: if the download succeeded, we do not download again, and if extraction succeeded, we do not pay a parser again.
Under the hood, a router splits each ingest before any workflow starts. Files are grouped by type and size into workflows, and anything over 50 MB gets a workflow of its own, so one huge PDF cannot stall a batch of small ones. Inside a workflow, each stage fans out per file, and one bad file does not fail the batch.
We were just as deliberate about what not to retry. A single error classifier decides what is retryable. A corrupt, oversized or unsupported file, or an API rejecting a request outright, is marked non-retryable and fails once, while rate limits, timeouts and network errors retry with backoff. We do not spend Worker time or paid API calls on work that cannot succeed.
We also pulled parsing out of the monolith into its own service, which runs on Temporal as well. Each file type maps to an ordered chain of parsers, each suited to different kinds of documents. Every parser in the chain runs as its own child workflow, so when one provider fails, the file moves to the next. Results are cached by a hash of the file's contents and the parser settings, and the parsing service checks that cache before it fetches anything, so an unchanged file is not parsed twice.
Hien: Some external parsers take minutes. How do you wait for them?
Frank: Without holding a Worker. The parsing workflow submits the job along with a correlation ID that points back to that exact workflow. Then it parks on a durable wait with a timeout, which costs nothing while the provider works and survives restarts. When the job finishes, the provider calls our webhook endpoint. We validate the request, use the correlation ID to find the waiting workflow, and signal it. The signal carries only a small status, not the document. The workflow wakes up, confirms the status, and fetches the full result in an Activity. If the webhook never arrives, the timeout fires and the chain moves on to the next parser.
Hien: How did Temporal change debugging and scaling for the engineering team?
Frank: The Temporal UI shows exactly how far an execution got and where it stopped, so debugging no longer means adding logging everywhere. If our own code hits an edge case extracting a CSV file, I can see the file, the stage, and the error, understand what happened, and fix it. Every workflow is also tagged with search attributes for the organization, knowledge base and file, so when a customer asks about one document, I can pull up every workflow that ever touched it.
Separating the work also gave us a clean way to scale horizontally. Downloads mostly wait on the network, while parsing a large PDF needs a lot of memory, so parsing runs in its own service with its own Temporal Workers and task queues. We size the parsing fleet for memory-heavy work and the main backend for I/O-heavy work, and scale one without scaling the other. Nothing heavy runs in the application's event loop; it all happens in background Workers, so the product stays responsive during large ingests. We also keep workflow histories lean: files and parsed documents stay in object storage, and Activities pass signed URLs to them instead.
Hien: How do you keep indexing fast when many customers ingest at once?
Frank: New indexing work is admitted in a controlled way, so a burst of activity in one place cannot starve everything else. A workflow waiting its turn sleeps on a durable timer instead of holding a Worker, and we use Temporal task priorities so in-flight work is dispatched ahead of newly enqueued work.
The next step is making the Workers themselves smarter about workload. Our demand is spiky, and keeping a large fleet of Workers running just in case a spike arrives is expensive. Our infra team, led by Lokesh Challagulla, is tuning how Workers scale with demand.
Customer complaints about indexing stopped#
Hien: Once the new pipeline was running, what changed?
Frank: They stopped complaining. We went from an indexing error rate of approximately 20 percent to less than 0.1 percent. Indexing used to come up with customers all the time. Then it went quiet.
It was not only Temporal, and it was not only me. Gabriel Marmelstein drove much of the parsing service optimization, Gabriel Napadenschi contributed optimizations across the pipeline, and Erik Wrede optimized how we use external parsers such as LlamaParse. With Temporal preserving the work that had succeeded and retrying at the right level, those changes together cut parsing costs to roughly one-tenth.
Solving the problem changed more than the pipeline#
Hien: What did solving this problem mean to you personally?
Frank: Solving a hard problem like this builds credibility, and the same solution could be exported to other parts of the product. About six months after I joined, I was promoted to CTO of StackAI. A few months later, StackAI was acquired by Asana.
Hien: What has it been like working with the Temporal team?
Frank: Temporal is very useful for us, and the people there are always willing to help debug a problem or brainstorm a solution. It is now one of the main components of the StackAI stack.
The work before the model runs#
Most of the hard engineering in an AI product happens before a model ever runs: downloading, parsing, chunking and indexing thousands of files that were never designed to be read by a machine. By making each of those steps durable, observable and retryable at exactly the right level, the team turned indexing into infrastructure nobody has to think about, and freed itself to build on top of it.
One of the clearest signs of the change was what Frank stopped hearing.
"Once the engineering problem is solved, it becomes very quiet. And that's a good thing."