The customer#
Cogent integrates with the security, IT, and business systems a customer already runs. We combine that data into a current model of assets, vulnerabilities, ownership, and business context. People and agents use that model to decide what to fix and take action.
That only works if the data layer can keep up. Enterprises are understaffed and already dealing with millions, sometimes hundreds of millions, of findings from cloud providers and vulnerability scanners. Tools like Slack, Confluence, and Jira provide organizational context. The product has to bring these disconnected sources together into one current model of the environment and keep it updated.
The challenge#
Our extraction Workflows ran on Temporal, and other Temporal Workflows coordinated Spark jobs that transformed and loaded the data. Engineers manually created Schedules for both. An automated email let us know when a customer's integration credentials were ready to use. An engineer would then create separate extraction Schedules for each type of data we needed from each integration, such as assets and vulnerabilities, and a Schedule for the tenant's processing pipeline. Even when we already supported the integration, someone on our team had to do this setup.
We also had to align the Schedules. We would see roughly how long the extractions took, then offset them from processing to minimize the time between the two. Extraction times varied by source and data volume, so we were always working with an estimate. We wanted processing to start as soon as the extractions finished.
The Workflows themselves already worked. We needed to automate the setup and the decisions we were making around them: start extraction when credentials are added, wait for the data, start processing, and keep doing this as the customer adds integrations and their data changes.
The solution#
We were cautious about introducing a long-lived entity Workflow to coordinate this work. We needed to manage its state and ensure we could restart it if it terminated. The individual extraction and processing Workflows were straightforward, whereas the coordinating Workflow's responsibilities were fuzzy at the beginning. We manually created and adjusted Schedules for a while, saw the same coordination problems come up, and eventually understood the behavior we needed well enough to automate it.
Once we did, we added a coordinator Workflow for each tenant to manage extraction and processing together. It follows the entity-workflow pattern, tracking current and pending work across runs.
Our integration service already validated customer credentials using the same API clients as our extractors. We extended it to create and update a single recurring Schedule for the tenant's coordinator. When credentials are added or updated, it also signals the coordinator to extract data from that integration API. It uses Temporal's signal-with-start operation, which starts the coordinator if it isn't already running. The recurring Schedule starts a small trigger Workflow that uses the same signal-with-start call.
The coordinator starts the existing extraction Workflows as children and tracks their completion. They can run in parallel and take different amounts of time. Once the extraction phase finishes, the coordinator checks the results and starts the processing Workflow. If all extractions in the run failed, it skips processing.
What happens if a customer adds another integration while this is going on? The coordinator receives a signal and handles it according to the current phase. During extraction, the new extraction can join the current run. If processing has already started, we can begin extracting the new data for the next run while the current processing continues. Extraction can also overlap with processing when triggered by the recurring Schedule. If the next extraction is due while processing is still running, the coordinator starts collecting data for the next run.
Temporal keeps the coordinator's state durable across Worker restarts. These runs can take hours, and the Worker spends much of that time waiting for other work to finish. When there's no more work, the coordinator exits; the next signal-with-start call can start it again using the same Workflow ID.
We also pin Workflow executions to the Worker version they started on. We can deploy new code while existing runs finish on the old version. The coordinator uses Continue-As-New to carry its state into another execution when needed, including to pick up a new Worker version at a safe boundary. This keeps version branches out of our Workflow code and makes changes easier to reason about, especially for engineers new to Temporal.
The results#
Each tenant went from many manually managed Schedules to one, created and maintained by the integration service. Engineers no longer set up Schedules when a credential-notification email arrives. That work was time-consuming and easy to get wrong, and it grew with every new integration and tenant.
We also stopped adjusting Schedule offsets around estimated extraction durations. The coordinator observes extraction completion and starts processing when the run is ready. If an extraction takes longer than usual, processing just starts later.
The takeaways#
Two lessons stand out.
Model ongoing coordination around a business entity. For us, that was the tenant. Customers add integrations while their data is still being processed, and scheduled updates keep arriving. A long-running coordinator for each tenant handles that ongoing work by running the same extraction and processing Workflows we used to schedule by hand.
Make pipeline dependencies explicit early. The coordinator made one dependency explicit: processing waits for extraction. Inside the processing pipeline, the dependencies between stages, meaning which Spark jobs read data that earlier jobs produce, still live only in the order of our Workflow code. Writing the pipeline this way was straightforward at first, but it has become harder to extend as it has grown. After a failure, we can skip completed stages, but choosing what to rerun still requires knowing which downstream stages need to be recomputed and which results can be reused. If we were starting again, we would make those dependencies explicit from the beginning.
