| This is a guest post from Praveen K B, Founding Engineer at Parseable |
A customer can complete onboarding and still be unhappy with the experience. A Workflow’s Completed status tells you it finished, but not where it spent its time. Onboarding might have taken five minutes or 50 because an Activity waited for a Worker, an identity provider timed out, or a Task Queue received Tasks faster than Workers could process them.
We built a simple onboarding Workflow that runs on Workers connected to Temporal Cloud and sent its telemetry to Parseable. This gave us two views of the process: Temporal SDK metrics showing Worker and Task Queue activity, and lifecycle events and traces showing what happened within an individual Workflow Execution.
TL;DR: Combine the metrics from the Temporal SDK with lifecycle events and traces to analyze when Activities started and how much time retries added to the process.
The validation setup#
Our test Workflow verifies a customer's identity and opens an account. We configured verifyIdentity to fail twice and succeed on its third attempt. That controlled failure let us check retry records, Activity duration, Workflow completion, custom events and trace relationships.
The Worker sends telemetry along two paths:
Workflow telemetry follows separate paths for lifecycle data, traces and SDK metrics before joining in Parseable.
The Parseable Temporal plugin writes Workflow and Activity lifecycle records to temporal-logs and spans to temporal-traces. The Temporal SDK exposes Prometheus metrics at the Worker. Prometheus scrapes that endpoint and writes the samples to temporal-metrics in Parseable.
For TypeScript, the following setup enables SDK metrics and configures the Parseable plugin:
import { Runtime, Worker } from '@temporalio/worker';
import { ParseablePlugin } from '@parseable/temporal';Runtime.install({
telemetryOptions: {
metrics: {
prometheus: { bindAddress: '0.0.0.0:9464' },
},
},
});const worker = await Worker.create({
connection,
namespace: process.env.TEMPORAL_NAMESPACE,
taskQueue: 'onboarding',
workflowsPath: require.resolve('./workflows'),
activities,
plugins: [
new ParseablePlugin({
serviceName: 'onboarding-worker',
endpoint: process.env.PARSEABLE_URL,
auth: {
username: process.env.PARSEABLE_USERNAME,
password: process.env.PARSEABLE_PASSWORD,
},
logs: { stream: 'temporal-logs' },
traces: { stream: 'temporal-traces' },
}),
],
});await worker.run();Workflow code can emit domain milestones through the plugin's replay-safe workflowEvent() helper. Our test emitted onboarding.started, identity.verified and account.provisioned. The plugin prevents Workflow replays from duplicating those records.
Five signals for Workflow health#
1. Schedule-to-Start latency#
This latency indicates the interval between Temporal scheduling a Task on a Task Queue and it being picked up by a Worker for execution. The metrics temporal_activity_schedule_to_start_latency and temporal_workflow_task_schedule_to_start_latency can be broken down by namespace and task_queue.
It is essential to examine this latency in combination with host-level utilization and slots. Increasing latency and high CPU or memory utilization point to Worker saturation. Increasing latency and idle hosts indicate issues with pollers, Worker connectivity, or slot configuration. In the context of onboarding, this metric helps distinguish time spent waiting in the Task Queue from time spent executing verifyIdentity.
2. Worker slot pressure#
Temporal Workers use execution slots to control how many Activity Tasks, Local Activity Tasks and Workflow Tasks they can process concurrently. It is useful to track the temporal_worker_task_slots_available and temporal_worker_task_slots_used metrics. The worker_type label differentiates between Activity and Workflow Workers.
Low temporal_worker_task_slots_available combined with rising Schedule-to-Start latency indicates an oversubscribed Worker pool. High available-slot counts combined with rising latency point towards poller or Worker connectivity issues.
3. Activity retries#
Retries may recover an Activity while adding customer-visible delay. The Parseable plugin writes lifecycle records for each Activity attempt with fields such as activity_name, activity_id, attempt, status and duration_ms. This query groups Activity outcomes and shows the highest attempt number:
SELECT
activity_name,
status,
COUNT(*) AS attempts,
MAX(attempt) AS highest_attempt,
AVG(duration_ms) AS avg_attempt_duration_ms
FROM "temporal-logs"
WHERE type = 'activity'
AND status IN ('completed', 'failed')
AND p_timestamp > NOW() - INTERVAL '1 hour'
GROUP BY activity_name, status
ORDER BY highest_attempt DESC;4. Workflow completion time#
The plugin records duration_ms when a Workflow Run completes or fails. You can group the events by workflow_name to get the p50 and p95 duration.
A Workflow that uses Continue-As-New creates a new Run with a different ID. You can track the progress of the business process across both Runs in a single correlation rule. For the onboarding process, the application team may want to capture the duration of each Workflow Run as well as the customer-perceived time from the initial request to delivery of the welcome email.
5. Task Queue backlog#
Temporal exposes ApproximateBacklogCount, ApproximateBacklogAge, TasksAddRate and TasksDispatchRate for Task Queues. ApproximateBacklogAge is often the best first metric to look at because it indicates the approximate age of the Task at the head of the backlog.
The values are approximate because an expired Task can remain in the Task Queue until it reaches the front. An idle Task Queue may be unloaded from memory, causing ApproximateBacklogCount to report zero until a Worker poll, new Task or Task Queue API call reloads it. Pair these values with Schedule-to-Start latency and Worker slot metrics. Temporal's Task Queue performance guide describes the limits.
Following one onboarding run#
We ran the validation Workflow against Temporal Cloud and inspected its records in Parseable. The verifyIdentity Activity produced failed records for attempts one and two, then a completed record for attempt three. The Workflow completed after openAccount, and all three domain milestones appeared in temporal-logs.
A controlled retry sequence verifies that Parseable preserves each Activity attempt and its place in the Workflow.
We used one query to reconstruct the execution:
SELECT
timestamp,
type,
status,
activity_name,
attempt,
duration_ms,
event_name,
error
FROM "temporal-logs"
WHERE workflow_id = '<workflow_id>' AND run_id = '<run_id>'
ORDER BY timestamp;The trace view showed RunWorkflow:onboardingWorkflow, StartActivity:verifyIdentity and RunActivity:verifyIdentity spans. It also showed the corresponding spans for openAccount. Logs provided searchable outcomes; the trace provided timing and parent-child relationships.
This was a controlled validation run, not a production benchmark. It confirmed that the integration preserved the identifiers and attempted data needed for an investigation.
A dashboard that leads to action#
Our starting dashboard uses six panels:
- Activity schedule-to-start latency by Task Queue
- Workflow Task schedule-to-start latency by Task Queue
- Available and used Worker slots by Worker type
- Activity failed attempts by Activity name
- p50 and p95 completed Workflow Run duration by Workflow name
- Recent failed Workflow and Activity records
Each alert should suggest a first check. When Schedule-to-Start latency increases, compare Worker slots, host utilization and poller behaviour. When retries increase, group failed attempts by activity_name and inspect the error field. Set thresholds based on workload expectations rather than applying one latency threshold to every Task Queue.
Use low-cardinality fields such as workflow_name, activity_name, task_queue, namespace, status and service_name for dashboard groups. Keep workflow_id, run_id and activity_id in lifecycle records and traces for targeted investigation. This split lets an operator move from a fleet-level symptom to one execution without turning each identifier into a metric label.
Parseable supports these views through SQL queries, dashboards and threshold alerts. Its Temporal integration guide documents the shared schema for TypeScript, Python and Java plugins.
What we learned#
The validation drew four practical lessons:
- Completion status alone doesn’t explain where a Workflow spent its time. Queue metrics, lifecycle records and traces help fill in the picture.
- Retry analysis needs records for individual Activity attempts. Recording only the outcome hides failed attempts that later recovered and the time they consumed.
- Check metric names and units before setting dashboard thresholds, especially when comparing Workers that use different SDKs.
- Fleet dashboards and execution identifiers serve different purposes. Use low-cardinality labels for monitoring and Workflow and Run IDs to investigate individual executions.
Temporal persists durable application state and execution history. SDK metrics expose Worker and Task Queue activity. Parseable adds queryable lifecycle records and traces that show whether work completed as expected or retries added delay.