This is a guest post written by Houman Kargaran, Engineering Lead at ANZ Bank
The full working code is available at https://github.com/houmanka/PII-Compliant-RAG.
In my first article, “Why Temporal for a data pipeline?”, I made the case for building on Temporal. Now it’s time to start on the actual requirements.
The first requirement is simple: a Pub/Sub message should trigger the application.
This is where the story starts. I’ll tell you the story of an engineer’s life before Temporal’s Standalone Activities.
Story time#
Once upon a time, there was an engineer who needed to process messages from a queue. Simple enough — or so it seemed.
The first version looked reasonable: run a consumer loop, read the message, process it, commit the offset. (The engineer was filled with joy.) But then reality arrived.
“What happens when the processing crashes halfway through? You need retry logic,” the tech lead asked. (“Easy peasy,” the engineer replied.)
The tech lead kept asking:
- What kind of retry?
- Exponential backoff?
- Fixed interval?
- How many attempts before you give up?
- And when you do give up, where does the message go — a dead-letter queue?
- A database?
- Who monitors that?
So you write the retry logic. (The engineer started to question his life choices around this point.)
Then you need to track state.
Did this message actually complete, or did the Worker crash after step 2 of 6? So you add a status table to the database.
Now you need something to watch that table and pick up stuck jobs. So you write a reconciliation job.
Now you need to monitor the reconciliation job. Three weeks later, you have a distributed state machine... (The engineer had enough.) As he reached for the delete button, the tech lead said:
“Have you thought about visibility? A message comes in, disappears into the system, and you have no idea if it succeeded, failed, or is silently stuck somewhere in the middle.” (That was it. The engineer left to fight another day.)
This was the world before Temporal’s Standalone Activities.
Enter Standalone Activities#
A Standalone Activity is a single Activity run as a top-level execution, kicked off straight from a Client with no Workflow around it. The Temporal docs spell out the full definition, including why that means fewer Billable Actions in Temporal Cloud and lower latency on short-lived Activity Executions.
I cried a little bit with joy when I got here. The power and freedom of Standalone Activities are an absolute God send.
A Standalone Activity acts as a durable bridge between Pub/Sub and Temporal.
The result of using a Standalone Activity is that the event handler stays thin. It receives the message, hands it off, and moves on. Temporal is going to look after the durability.
At the time of writing, Standalone Activities are in Public Preview. This pipeline is demo code, so using a Preview feature is the right call here.
Here is the Standalone Activity:
@activity.defn
async def ingestion_handler(arg: FileInput) -> str:
client = activity.client()
handle = await client.start_workflow(
"ComplaintWorkflow",
arg,
id=f"complaint-workflow-{arg.path}",
task_queue="INGESTION_QUEUE",
id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING,
id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE,
)
return handle.idThe Workflow ID is derived from the file path, not a random UUID. This matters because ingestion_handler is a Temporal Activity. If it fails and retries, a fresh UUID would start a second ComplaintWorkflow for the same file, silently doubling the work. With a deterministic ID, every retry uses the same Workflow ID, and the configured ID policies prevent duplicate Temporal starts. The file is the unit of work, so the file path is the right identity.
And here is how the subscriber calls it:
await client.execute_activity(
ingestion_handler,
args=[FileInput(provider=payload["provider"], path=payload["path"], event_type=payload["eventType"])],
id=f"ingest:{payload['path']}",
task_queue="INGESTION_QUEUE",
start_to_close_timeout=timedelta(seconds=100),
id_conflict_policy=ActivityIDConflictPolicy.USE_EXISTING,
)The id must be unique per message. This is the key thing to remember. A hardcoded string means every Pub/Sub message shares one Activity ID, collapsing all ingestion events into one entry in the Temporal UI and making individual file executions invisible.
Setting id_conflict_policy=ActivityIDConflictPolicy.USE_EXISTING means that if Pub/Sub redelivers the message while the Activity is still running, Temporal attaches to the in-flight execution rather than erroring or starting a second ingestion of the same file. Each file ingestion is independently visible and traceable in the Temporal UI by path.
The subscriber does not start the Workflow directly. It runs the Standalone Activity and acknowledges the Pub/Sub message after that Activity returns.
From this point on, the job of our event handler is done. We can ACK back and consume the next message. The Temporal Workflow looks after the rest.
Other use cases for Standalone Activities#
Since I am a huge fan of Standalone Activities and I think they are the best invention since sliced bread, I am going to give you more use cases.
Any time you need to reliably execute a single function in response to an external event, with built-in retries and timeouts but without the overhead of a full Workflow, this is the right tool. The keyword is a single function: isolate the thing you want to do. It is not a Workflow. Common examples include sending an email, processing a webhook, or syncing data from an external system.
In our case, it bridges a Pub/Sub event to a durable Workflow, keeping the event handler completely decoupled from the processing logic.
What’s next#
This article covered how we bridge Pub/Sub to Temporal using a Standalone Activity and why the event handler stays thin. The next article goes deeper into the first Activity that runs inside the Workflow, IngestFileActivity, and how its single-Activity design enforces the PII boundary at the architectural level.