How Rapidflare built a million+ document ingestion pipeline for Agents on Temporal

Guest post by Dipkumar Patel, founding engineer, and Vasanth Asokan, CTO and co-founder of Rapidflare.

At Rapidflare, we build Agents for technical sales teams in the electronics and semiconductor industry, where product information and technical literature are dense, voluminous and organized in complex taxonomies. Our customer’s Agent queries and tasks require precision and comprehensiveness. A confidently wrong or partial answer is worse than no answer at all.

• A single electronics product datasheet can run to hundreds of pages of specs, tables, and compliance data.

• A single product may have supporting material in the form of user guides, application notes, support articles, development tool manuals, software packages, schematics, errata, and more.

• A single customer may have thousands of such products. Their knowledge base can contain hundreds of thousands of such documents spread across Confluence, SharePoint, Google Drive, Salesforce Knowledge, and other systems.

Pre-structured knowledge is the heart of our Agent#

Now imagine a query like this:

“For a four-channel SerDes design at 10.3125 Gbps NRZ, with two channels running DFE for backplane equalization, what’s a reasonable total power budget across all four lanes?”

A high-quality, fast, and complete response to a query like the one above requires us to first identify, ingest, and pre-structure technical knowledge into proprietary knowledge graph formats. We also need to be deeply confident that we’ve processed and organized every single piece of literature on this topic from the customer’s vast corpus of knowledge through our document ingestion pipeline.

Our Agent harness relies on proprietary techniques for retrieving and reasoning over the most relevant nodes of customer documentation. This may happen dozens of times in a single agentic turn, sometimes more. The floor for answer quality starts at the ceiling of our content ingestion capability and reliability. If a content source is crawled incompletely, chunked badly, or left to go stale, no amount of agentic intelligence can save the day. The model just sounds confident about the wrong thing.

So for us, ingestion is not a preprocessing step you run once and forget. It is the system.

What a good ingestion pipeline has to do#

On paper a document ingestion pipeline may look trivial:

Crawl source -> Extract text -> Chunk -> Embed -> Store in vector + keyword stores

One customer can have a small sitemap consisting of a few dozen pages. A much larger customer may have hundreds of thousands of PDFs and datasets. The same pipeline has to handle both, and when it fails to ingest content from a source, ultimately the customer just loses trust in our Agent.

A serious ingestion pipeline for Agents has to be:

• Durable. Runs take hours. A crash at hour four cannot mean starting over from zero.

• Stateful. The pipeline needs to know which documents succeeded, failed, or were skipped, and where to resume.

• Concurrency-controlled. The processing stages for each document involve expensive, rate-limited downstream infrastructure calls (PDF extraction, VLMs, LLMs, embeddings, etc.). Unbounded fan-out makes things slower, not faster.

• Observable. When we process thousands of documents across distributed workers, we need a reliable way to trace one failure back to one document or have provenance for one piece of fully processed content back to every processing step.

• Approval-gated. “Successfully processed” is not the same as “safe to serve.” Freshly indexed data may need a review (human or automated) before it goes live."

• Source-agnostic. We support 20+ source types powered by enterprise integrations. Adding the twenty-first should not mean redesigning the pipeline and should take very little effort to add.

• Fresh. Customer docs change. The pipeline has to re-ingest on an automatic schedule or upon change triggers.

With this bar in mind, let’s review the requirements in more detail and the architectural or design choices we made in order to meet them.

Where ingestion gets hard, and why we chose Temporal#

Things break first when you scale a pipeline like this up:

Rate limits. Each document triggers LLM calls for image and table description and summarization, embedding API calls, and writes to multiple data stores (Turbopuffer, PostgreSQL, ElasticSearch). The expensive, rate-limited resources are downstream, not in our own code. If we simply fanned out everything all at once and if some of those resource-limited providers started returning 429s, the whole run would get slower the harder it pushed.

Resource exhaustion. Worker pools are finite. Fan out too aggressively and you get queueing, memory pressure, and cascading timeouts.

The long pole. Even with fixed batches, one massive datasheet can hold an entire batch of slots while other work sits idle.

Individual stage failure. On top of that, every stage fails independently: PDF extractors crash on malformed files, LLM calls hit 503s, crawlers run for hours and die in the middle.

With all these problems, we needed durable, resumable, multi-hour orchestration with retries, signals, and state checkpointing.

The requirements above left no real choice: we needed Durable Execution before we wrote a line of pipeline code. We did evaluate the alternatives first, including Apache Airflow, Windmill, and Hatchet. We also considered simply running end-to-end batch jobs on a single machine. Temporal won on the combination we cared about: durable, resumable Workflows with first-class Retry Policies. We made a very early decision to use Temporal casting aside any doubts we may have had in investing in a complex framework and system, as we knew we’d hit problems like this. It would have been harder to retrofit Temporal into a cobbled-together ingestion pipeline. We definitely faced a learning curve with Temporal, but it paid off over time.

The first architecture we built on top of it still fanned out every document at once, and a customer with a single 1,000,000-document source found both of our wrong assumptions.

The pipeline, stage by stage#

A user triggers a run, and it moves through three phases: 1) connect, crawl and stage; 2) process (extract, clean, chunk, cross link, summarize, embed) every document; then 3) approve and promote. A separate status Worker tracks progress the whole time, and the whole thing is source-agnostic. Every crawler produces a single “staged document” artifact, so everything downstream of the crawl gets an identical input document shape, whether the source is a sitemap, Google Drive, or Confluence.

We run three Temporal Workers in one process, each on its own Task Queue:

• Ingestion: Crawling, extraction, embedding, indexing, and higher concurrency.

• Enrichment: Post-ingestion summarization and tagging, with lower concurrency).

• Status sync: Progress persistence, with lower concurrency.

The isolation boundary is the Task Queue and its concurrency budget. A burst of extraction tasks can never starve the status update tasks that drive the user UI.

Stage 1: Crawl and stage the source#

The staging Activity runs a source-specific crawler (Sitemap, Google Drive, or Confluence) Staging can run for hours on a large document library. It is the one of the longest single pieces of work in the pipeline, and three Temporal features that help us.

First, Heartbeats. Temporal uses Heartbeats to tell “I am still working” apart from “the Worker crashed or failed.” If the Activity stops heartbeating, Temporal reschedules it on another Worker, which for a crawler means starting the source over. So we send Heartbeats at milestones, such as "component_datasheets: fetched 500 datasheets", instead of "heartbeat #294”.

Second, a deliberate maximum_attempts=1 on this Activity. We tell Temporal not to retry the thing because our crawlers already have a retry mechanism, rate limits, and the ability to handle transient errors internally with their progress intact. If Temporal retried on top of that, it would discard hours of crawler progress and start from zero. The rule we settled on: retry at the layer that is cheapest to retry, which is inside the crawler, not at the Activity level.

Third, Cloud Storage as a data bus, because a crawl has to fetch thousands of documents and its metadata, far past Temporal’s payload limits. The staging Activity writes results to a bucket and returns only a lightweight reference. Downstream Activities load one page at a time. This also solves distributed execution: an Activity on one Cloud Run instance and one on another share state through the bucket, not through Event History.

Temporal doing the work here: Activity Heartbeats, per-Activity Retry Policy (maximum_attempts), and Activities as the isolation boundary for external I/O.

Stage 2: Fan out and process every document#

This is where the failure to ingest a million documents drove most of our design decisions. Instead of starting every document at once, the Parent Workflow keeps exactly N Child Workflows in flight via a sliding window mechanism. The moment one finishes, the next starts, keeping all Workers busy.

@workflow.signal
async def on_doc_complete(self, result: CompletionResult):
    self._signals.append(result)

# Inside the main workflow run:
for doc in page.docs:
    await workflow.wait_condition(
        lambda: len(self._active) < params.window_size
    )
    self._process_signals()

    await workflow.start_child_workflow(
        ProcessDocWorkflow.run, doc,
        id=f"{workflow.info().workflow_id}/doc/{doc.id}",
        parent_close_policy=ParentClosePolicy.ABANDON,
    )
    self._active.add(doc.id)

A few Temporal mechanics make this work. workflow.wait_condition blocks without polling, re-evaluating only after each completion Signal arrives. Children report back with @workflow.signal rather than the parent awaiting their results, so the parent stays in control of the window. The Child Workflow ID is deterministic (parent ID plus document ID), so a duplicate start becomes a predictable ID conflict instead of two Workers racing on the same document. And ParentClosePolicy.ABANDON lets children outlive a parent restart.

Managing state with Continue-as-New. Tracking 1,000,000 document IDs in Workflow state blows past Temporal’s history limits. So we split documents into pages during the crawl and keep only a page cursor (a single integer), the in-flight IDs (at most N), and success/failure counters. That is constant-size state at any scale. At each page boundary, if Temporal signals that history is getting large, we checkpoint and Continue-as-New. In-flight children survive the restart (thanks to ABANDON policy) and Signal their results to the fresh parent instance by Workflow ID. A crash now costs just the current page, not the whole run.

The PDF sub-case. A single datasheet can be hundreds of pages, so large PDFs get split and their chunks fan out with a simpler batch-and-wait pattern. Chunks are uniform in size and bounded in number, so the long-pole problem barely applies:

handles = [
    await workflow.start_child_workflow(ProcessChunkWorkflow.run, small_section)
    for small_section in full_pdf
]
results = await asyncio.gather(*[h.result() for h in handles])

Every per-document path runs document extraction, OCR, LLM, VLM (image and table description, summarization), and embedding generation. The sliding window mechanism is extremely helpful, because many of these calls are rate-limited by their providers, which means we can’t process all documents in parallel.

Temporal doing the work here: Child Workflows, Signals, wait_condition, deterministic Workflow IDs, ParentClosePolicy.ABANDON, Continue-as-New, futures via asyncio.gather, Activities isolating nondeterministic model calls.

Stage 3: Approve, then promote with zero downtime#

Every ingestion has to go through an approval stage, either manual or automated, depending on its configuration.

The Workflow blocks on wait_condition until it receives an approval or rejection Signal. Temporal makes this durable: the Workflow can wait for days, and if the Worker restarts, it resumes exactly where it left off.

Manual review doesn’t scale to fifty sources on weekly schedules, so we have set of policy that user can choose:

• MANUAL: Always waits for human review (notifies the user through channels such as Slack or email).

• AUTO_APPROVE: Automatically approve the ingestion after strict sanity checks.

• STRICT: Auto-approves only if every document succeeds.

One rule overrides all three: zero documents in the staging Activity means automatic rejection. A source that silently empties its index is worse than one that fails loudly.

On approval, we promote with a zero-downtime swap. Old and new data coexist, each tagged by run ID. We flip the live pointer to the new run ID and retire the old one. On rejection or timeout, we discard the new copy and leave live data untouched.

Temporal doing the work here: Signals, durable wait_condition (a Workflow that waits days for a human), and Workflow timeouts for auto-reject.

Stage 4: Operating it (cancellation, observability, Scheduling)#

Cancellation of ongoing ingestion. Users might want to cancel ingestion at any point, but the right response depends on which stage we’re in. We check a cancellation flag at lifecycle boundaries. For multi-hour staging Activities, we use Heartbeats for cancellation so a user doesn’t wait 40 minutes for the crawl to notice the cancellation Signal. If a user triggers cancellation during staging, we cancel the Activity on the next Heartbeat. During the fan-out phase, we cancel all Child Workflows in the current sliding window and clean up any data already ingested. The same logic applies during the approval stage. This took us quite a while to get right.

Observability of ingestion through a dedicated status Worker. Once Ingestion is triggered by a user and the Temporal Workflow starts, the user needs live UI updates on its progress. Each Activity emits its progress to status Activity, which gets picked up by the status Worker (on its own task queue) and writes progress to the database. The UI reads from the database. This keeps a saturated ingestion Worker from ever blocking progress updates. Every log line also carries correlation IDs at three levels (Workflow, document, Worker instance) so one document’s failure can be traced across restarts and instances.

Recurring ingestion through Temporal Schedules. Customer docs change, so most sources re-ingest on a cadence. We use Temporal’s native Schedules instead of an external cron, which gives us durability, pause/unpause, and backfill for free. The Schedule fires a thin wrapper Workflow that loads fresh source config before dispatching, because a schedule from three months ago should not use three-month-old credentials. It also guards overlap with ScheduleOverlapPolicy.SKIP, so active runs never collide with new ones.

Temporal doing the work here: Activity cancellation (ActivityCancellationType.TRY_CANCEL), asyncio.shield for cleanup, the Query API for the read model, Temporal Schedules with ScheduleOverlapPolicy.SKIP, and developer-namespaced Task Queues for local isolation.

Takeaways#

If your document ingestion jobs are short and idempotent, you may not need any of this. The equation changes the moment a run gets long enough that “just run it again” means throwing away hours of work. For anyone building a content or document ingestion pipeline, here are our summarized takeaways:

• Retry at the cheapest layer. Sometimes that means instructing Temporal not to retry and allowing the component to handle retries internally. Temporal retries have an associated cost, so they should be used deliberately. Review each Workflow and Activity, and configure Retry Policies according to their individual needs.

• Separate Workers by concern. Isolate Workers based on their responsibilities and scaling patterns. Ingestion, enrichment, and status sync on their own Task Queues so they cannot starve each other and one can be scaled independently.

• Heartbeat at each milestone inside a long Activity. In a long Activity, Heartbeats do more than keep Temporal updated on Worker status. They also make cancellation flow much cleaner.

• Use cloud storage as a data bus. Temporal Payloads have size limits. For any large Payload across an Activity or Workflow, it’s better to use cloud storage as intermediate bus exchange.

• Put the rate-limited work behind a fixed sliding window. If you’re fanning out your work across hundreds of Workers, some operations, like LLM/VLM calls, will be inherently rate-limited. We don’t want to start a retry storm. Use a sliding window pattern to avoid backlogging the queue.

• Add an approval stage before updating live data. Before applying changes to production systems, introduce either automated validation or human review. Workflow Signals and waiting states are helpful in implementing approval gates when needed.

• Prefer Temporal Schedules over external cron. For recurringWorkflows, built-in Scheduling provides better visibility, reliability, and operational control. We use Schedules extensively for automated ingestion Workflows.

Congratulations on making it this far! We hope our earned insights and takeaway summaries have been useful for you. We are having a blast building Agents for the electronics industry, and we’re just getting started. If our work or target customer domain sounds interesting to you, please reach out to the authors (Dipkumar Patel, Vasanth Asokan) or email us at hello@rapidflare.ai.