Dogfooding the Billable Actions metric: How granular observability improved our metering validation

As we ship the Billable Actions metric alongside the Billing API, we wanted to share some of our experience using this metric internally over the last few months. What started as an engineering initiative to give customers better usage visibility has become one of the most valuable tools for our own team, and an example of what continuous dogfooding looks like in practice.

Cost visibility without the details#

Temporal Cloud customers rely on the usage dashboard to understand their consumption. The dashboard breaks usage down by high-level categories: Workflows, Activities, Signals, and so on. This gives you a solid overview of Action consumption.

This view is useful, but it has limits. If you notice your Workflow Action count trending up, the dashboard tells you that it is growing, but not which Workflow is driving the growth. All Workflow Types are rolled up into a single bucket.

For customers trying to validate usage costs and answer questions like “Which Workflow Type is our biggest cost driver?” or “Can we optimize this one hot path?”, that missing granularity is a blocker. One Workflow Type could be responsible for the entire cost increase, but there would be no way to tell from the aggregate view alone.

What’s new: Per-type breakdowns via OpenMetrics#

With the release of the Billable Actions metric, customers can query the OpenMetrics endpoint and get a time series of billable Actions broken down by action_type and temporal_workflow_type.

For our internal validation dashboards, those same dimensions let us build views like the following.

Breakdown by Workflow Type#

SELECT
  toStartOfInterval(timestamp_interval, INTERVAL 3600 second) as "time",
  temporal_workflow_type,
  sum(count) as total_count
FROM
  "extobs"."temporal_cloud_v1_billable_action_count_all_activities"
WHERE
  $__timeFilter(timestamp_interval)
  AND temporal_account IN (${temporal_account:sqlstring})
  AND temporal_namespace IN (${temporal_namespace:sqlstring})
GROUP BY
  time,
  temporal_workflow_type
ORDER BY
  time ASC
${extobs_settings}

Breakdown by Action Type#

SELECT
  toStartOfInterval(timestamp_interval, INTERVAL 3600 second) as "time",
  action_type,
  sum(count) as total_count
FROM
  "extobs"."temporal_cloud_v1_billable_action_count_all_activities"
WHERE
  $__timeFilter(timestamp_interval)
  AND temporal_account IN (${temporal_account:sqlstring})
  AND temporal_namespace IN (${temporal_namespace:sqlstring})
  -- Optional Action Type breakdown filtering by Workflow Type
  -- AND temporal_workflow_type = 'MyWorkflowType'
GROUP BY
  time,
  action_type
ORDER BY
  time ASC
${extobs_settings}

Instead of “you ran X Workflow Actions this month,” you get “you ran X Actions from OrderProcessingWorkflow, Y Actions from PaymentReconciliationWorkflow, and Z Actions from NightlyBatchWorkflow.” You can see per-Workflow-Type meters, identify which Action Types are running hot, and build dashboards that surface optimization opportunities at the level where real decisions get made.

SELECT
  toStartOfInterval(timestamp_interval, INTERVAL 3600 second) as "time",
  temporal_workflow_type,
  action_type,
  sum(count) as total_count
FROM
  "extobs"."temporal_cloud_v1_billable_action_count_all_activities"
WHERE
  $__timeFilter(timestamp_interval)
  AND temporal_account IN (${temporal_account:sqlstring})
  AND temporal_namespace IN (${temporal_namespace:sqlstring})
  AND temporal_workflow_type in (
    'workflow.echo',
    'workflow.eager-workflow-start',
    'workflow.localactivity',
    'visibility-child-workflow'
  )
GROUP BY
  time,
  temporal_workflow_type,
  action_type
ORDER BY
  time ASC
${extobs_settings}

The problem it solved: Where dogfooding comes in#

Here’s where it gets interesting for Temporal internally.

On an ongoing basis, we validate our metering pipeline and make sure that what we meter is correct, down to the Action. As we release new features and introduce new billing rules, we need a way to continuously verify that the meters and bills we emit accurately represent the work a customer actually did.

Today, we validate metering through a dedicated canary account that generates a predictable, known workload across every billable Action. We query Temporal Server directly to pull raw Action counts, aggregate them, compare the totals against what our metering pipeline reports, and confirm that the two stay aligned. Customers can run this same validation on their own accounts if they want to audit usage independently.

When our validation pipeline flagged a discrepancy, we could see that the numbers did not line up, but pinpointing the cause was painful. Was the gap coming from a new Workflow Type we had recently shipped? A change in how a particular Activity was being counted? Or was it simply noise from server rollouts temporarily affecting the signal? With multiple potential root causes, debugging felt more like investigation than engineering.

The Billable Actions metric changed that.

Because the metric is emitted directly from the Temporal Cloud system that tracks metered customer usage, we can now put our validation scanner’s output side by side with the per-Workflow-Type and per-Action-Type metrics we already emit. When a discrepancy shows up, we do not have to guess. We can scope it to a specific Workflow Type, Action Type, or both, and investigate from there.

To be clear: our metering pipeline is correct. The Billable Actions metric gave us an extra layer of observability for validating and troubleshooting metering, which helps us keep up with the new features and billing rules we continue to ship. When you add multiple variables at once, the tool you need to debug cleanly is granularity. Aggregate counters hide the variables. Per-type metrics expose them.

The payoff#

The result has been a meaningful jump in confidence for our validation pipeline. Something that used to be nuanced and time-consuming to track is now readily visible. Troubleshooting discrepancies, which previously meant correlating across multiple dashboards and database queries, is now a matter of pulling up the right metric and following the signal to the specific Workflow Type, Action Type, or both.

And this is exactly the kind of process we want our customers to have. When we build tools that genuinely make our own day-to-day easier, that is a strong signal we are building something useful.

What’s next#

The Billable Actions metric is rolling out with the Billing API. If you are a Temporal Cloud customer, here is what you can expect to do with it:

• Usage visibility: See exactly which Workflow Types and Action Types drive your usage, not just category totals.
• Optimization: Identify high-volume Workflows that are good candidates for redesign or refactoring.
• Validation: Reconcile your expectations against what we are metering.
• Alerting: Build alerts on per-type Action counts to catch regressions or runaway Workflows early.

We’re excited to ship this one. It has already changed how we operate internally, and we are looking forward to seeing what customers do with it.