This blog from Xgrid, a Certified Temporal Cloud Partner, explores how field operations in construction, infrastructure, and facilities management can be modeled as a Durable Execution problem rather than a collection of fragile services.
Drawing from a real-world implementation, it shows how crew clock-ins, daily reporting, and document signing were rebuilt on Temporal using a single Workflow per shift as the durable source of truth.
The post highlights key patterns for handling offline-first operations, idempotency, compliance, retries, encryption, and scalable execution in unreliable field environments. It also demonstrates how this approach improved payroll accuracy to over 99%, virtually eliminated tool losses, simplified audits, and significantly reduced operational troubleshooting time.
Solving distributed state: surviving the 7:02 AM network drop#
A field technician taps "Clock in" at 7:02 AM. The app spins. Signal drops. They tap again. Forty minutes later, three queued requests drain to the server — possibly out of order, possibly after a clock-out from a different job.
Three questions payroll will eventually ask:
- What time did they actually start work?
- Did we record them once, twice, or not at all?
- Can you prove it?
These aren't edge cases. They're the product. Get them wrong and you don't get a bug report.
Instead you get a labor-compliance dispute or a tool missing from an active construction site, and a worker who may or may not have been present when the safety alarm went off.
This post is about how Xgrid worked with a construction company to rebuild crew clock-in/clock-out, daily work reporting, and document signing on Temporal for a large-scale field operations platform — and why Durable Execution turned out to be the right primitive for a business problem, not just a technical one.
Beyond the tech-glamorous use cases#
When Temporal comes up in engineering circles, the conversation usually runs toward AI agents, CI/CD orchestration, or high-throughput microservices. Those use cases are real and interesting.
However, there are deeper cases that Temporal can handle: workflows where the human executing the process is standing in a basement with no cell signal, carrying 40 pounds of tools, wearing gloves too thick to interact with a touchscreen, and operating under union rules that govern exactly when a break can start and end.
Field operations in construction, infrastructure, and facilities management are defined by four brutal realities:
- Long-running state. A shift is not a request/response. It opens in the morning, accumulates events such as clock-ins, safety questionnaires, mid-shift schedule changes, tool sign-outs, task completions, document signatures and closes at night. The "transaction" spans hours.
- Independently failing systems. A single clock-out can touch a SQL database, an ERP, a Windows file share holding signed PDFs, a headless browser rendering those PDFs, and an email notification path. Each is down or slow for different reasons at different times. None coordinate their failures.
- Unreliable edges. Underground workspaces, steel-framed buildings, and dense urban environments all kill GPS and interrupt cellular. A worker's device might be offline for two hours in the middle of a shift.
- The audit trail is the deliverable. "The database says 7:45" is not an acceptable answer when the technician's device captured the action at 7:02. The record of what was attempted, when, and why it landed where it did is itself a compliance artifact that clients inspect, auditors subpoena, and workers dispute.
The conventional response to these properties is sediment: status columns, cron sweepers, dedupe checks, retry wrappers, and reconciliation jobs — each added after a specific incident, none composing into anything you can reason about.
Temporal let us delete most of that sediment and replace it with one idea: the workflow is the source of truth for where a shift is, and it cannot lose that truth — not to a crash, a deploy, or a dropped connection.
Tayyab Mahmood - GM & VP Engineering at Xgrid
Architecture: one durable Workflow per shift#
The central design decision was simple to state and significant in its implications: each scheduled shift becomes exactly one Temporal Workflow, addressed by a deterministic ID derived from the schedule.
The mobile app never starts a second Workflow and never has to ask "did my last request land?" It issues Updates against the Workflow. Temporal enforces single-instance semantics at the platform level and not in application code.
The Workflow maintains the shift state in memory while it executes, and Temporal durably persists the Event History. If a Worker process redeploys mid-shift, Temporal replays that history to reconstruct the Workflow state and continue execution on another Worker. There is no "recover in-flight shifts" job because Workflow state is reconstructed automatically instead of being restored from an application database.
// SampleCode of Clock-in Clock-out workflow
async function clockInOutWorkflow({ scheduleID, employeeID }) { const queue = new RequestQueue(); // serialization + audit trail let state = STATES.PENDING; // durable shift state machine const results = {};
for (const step of Object.values(WORKFLOW_STEPS)) {
setHandler(
step.update,
async (payload) => queue.run(step, async (requestId) => { state = applyTransition(step, state, 'start');
const result = await processActivities(step.name, {
...payload, requestId });
state = result.success
? applyTransition(step, state, 'success', payload)
: applyTransition(step, state, 'failure');
return result;
}),
{ validator: (payload) => assertAccepted(step, queue, state, payload) }
);
}
await sleep(SHIFT_DURATION);
await condition(() => queue.isDrained());
return { success: 1, finalState: state, results, history: queue.snapshot() };
}The shape here matters more than any single line. The allowed state transitions and conflicts live in one readable place, expressed as ordinary JavaScript and version-controlled with the rest of the application.
How Temporal solved the hard problems#
1. Beacon-based location: replacing GPS with certainty#
The first version of the system tried GPS geofencing. It failed constantly: underground workspaces are invisible to satellites, steel-framed buildings cause 20–50 meter position drift, and urban canyon effects make precise location effectively impossible.
The replacement was Bluetooth Low Energy (BLE) beacons placed at site entrances, floor landings, and key work areas. When a technician walks within 5 meters of a beacon, the beacon ID is captured by their device and sent to the API. The Workflow validates the beacon ID against the scheduled job site and stores it with the clock-in. In this deployment, that provided a floor-level location without relying on GPS or requiring the worker to enter a location manually.
Accuracy improved from "somewhere within 30 meters" to "confirmed at Floor 5 of Site 123." Workers with gloved hands or full tool bags don't have to enter or confirm their location manually. The beacon reading provides location evidence, while Temporal orchestrates and persists the result.
2. Idempotency: making duplicates safe by design#
A flaky network will deliver the same intent more than once. Rather than hoping duplicates wouldn't happen, we made them safe at the platform level. At the workflow boundary, the schedule ID deterministically names the workflow. A duplicate "start shift" resolves to the existing run rather than creating a second one:
// Sample Code for starting temporal workflow
const handle = await client.workflow.start(clockInOutWorkflow, { workflowId: `cico-${scheduleID}-${process.env.NODE_ENV}`, taskQueue: TASK_QUEUE.CICO,
args: [{ scheduleID, employeeID }],
workflowIdConflictPolicy: WorkflowIdConflictPolicy.USE_EXISTING, workflowIdReusePolicy:
WorkflowIdReusePolicy.ALLOW_DUPLICATE_FAILED_ONLY,
});
The technician who taps "Clock in" three times produces one clock-in, not three.
This is the inversion that Durable Execution buys you: instead of writing defensive dedupe checks at every call site, idempotency becomes a property of the workflow's identity and its message-handling contract. An entire category of defensive code simply stops needing to exist.
Tayyab Mahmood - GM & VP Engineering at Xgrid
3. Offline-first resilience: Signal-With-Start#
The biggest challenge in harsh field environments is intermittent connectivity. In the legacy system, if a technician completed a task in a low-connectivity zone, the phone would try to send a "Task Complete" message after reconnecting, but it would fail because the server-side session wasn't expecting it.
Temporal's Signal-With-Start pattern solved this cleanly. The client issues a request that effectively tells Temporal: "deliver this signal to workflow X; if workflow X isn't running yet, start it, then deliver the signal." Even if the "start shift" event was never received due to connectivity loss, the first "Task Complete" signal retroactively initializes the workflow and records that task.
The mobile app buffers events locally and replays them in their original order when connectivity returns. Temporal durably records each accepted Signal in the Workflow History, and Workers poll Task Queues to process the resulting Workflow and Activity Tasks.
Temporal ensures that no matter what goes wrong with connectivity, servers, or infrastructure, the business logic is never compromised.
Tayyab Mahmood - GM & VP Engineering at Xgrid
4. Ordering and deduplication: one queue, one source of sequence#
Every Update such as clock-in, clock-out, questionnaire answers, mid-shift edits etc enters a single in-workflow request queue before touching state. The queue assigns sequence numbers, enforces ordering, and supersedes stale duplicates:
// Sample Code for request Queue
class RequestQueue {
async run(step, work) {
const requestId = ++this.seq;
if (step.latestWins) {
this.queue.forEach((r) => {
if (r.step === step.name && r.status ===
REQUEST_STATUS.WAITING) {
r.status = REQUEST_STATUS.SKIPPED;
}
});
}
const entry = { requestId, step: step.name, status:
REQUEST_STATUS.WAITING, at: Date.now() };
this.queue.push(entry);
/* A clock-out waits behind an in-flight clock-in even if a flaky client fired both within the same second. */
await condition(() => this.allPreviousSettled(entry));
if (entry.status === REQUEST_STATUS.SKIPPED) return { success: 0, code: 'skipped' };
entry.status = REQUEST_STATUS.IN_PROGRESS;
const result = await work(requestId);
entry.status = result.success ? REQUEST_STATUS.COMPLETED : REQUEST_STATUS.FAILED;
return result;
}
}The queue does double duty. Operationally, it prevents two near-simultaneous requests from racing into an inconsistent state. Forensically, snapshot() is the audit trail in every attempt, its sequence number, its status, and its timestamp, queryable for any shift on demand.
5. Compliance-as-code: validation before the write#
Temporal Update validators run before a request is admitted to workflow history. Invalid sequences become immediate, meaningful rejections and the app gets a clean error rather than a second execution and a later reconciliation:
// Sample Code for state conflict validation
function assertAccepted(step, queue, state, payload) { const conflict = queue.latestUnsettled(step.conflictsWith); if (conflict) {
throw ApplicationFailure.create({
message: step.conflictMessage, // e.g., "Your clock-in is already in progress."
nonRetryable: true,
details: [{ code: 'ALREADY_IN_PROGRESS', state }],
});
}
}Beyond deduplication, the workflow itself enforces safety rules. A worker cannot clock out if tools are still signed out to them and the workflow prompts a tool-return activity first. No task can begin until the safety briefing signal has been received. These rules aren't documented separately; they're encoded in the state machine, enforced deterministically for every site and every shift.
6. Retryable vs. non-retryable: classify by cause, not by symptom#
A timed-out file share should be retried until it comes back. A document that genuinely doesn't exist should fail immediately and loudly. Treating these the same is how systems either give up too early or hammer a dead dependency for hours:
// Sample Code handling activity errors & retries
const isRetryable = (error) =>
isDbConnectionError(error) ||
isFileShareError(error) ||
isBrowserError(error) ||
isErpConnectivityError(error) ||
isTransientMailError(error);
function handleActivityError(error, operation) {
if (isRetryable(error)) {
throw error; // Let Temporal's backoff policy own the retry. }
throw ApplicationFailure.nonRetryable(
`${operation} failed: ${error.message}`,
error.name ?? `${operation}Error`,
[{ code: 'NON_RETRYABLE', operation }]
);
}
// Retry policy is declarative — described once, enforced by the platform.
const clockInOutActivityPolicy = {
startToCloseTimeout: '2m',
retry: { initialInterval: '10s', backoffCoefficient: 2, maximumInterval: '12h' },
};If the PDF file share is unmounted for thirty seconds during a clock-out, the technician never sees an immediate service-down error. Temporal durably persists the operation and retries it with backoff according to the configured retry policy. When the file share becomes available again, the operation completes automatically without requiring the user intervention. To the technician, this is what durability feels like as a user. The application continues working through transient outages without disrupting the experience.
7. The payload is the compliance record#
The timestamp that belongs on a labor record is the moment the technician acted — not when the server happened to process the request. The mobile app captures that timestamp locally, on the device, the instant the button is tapped, even with no connectivity:
// Sample Code for sign Job Documents
async function signJobDocuments(input) {
const { schedule: { employeeID, jobID, clockInDate, clockOutDate }, jobDocuments } = input;
// The legally meaningful timestamp is what the device recorded — // NOT Date.now() at server-processing time.
const tx = await sequelize.transaction();
const browser = await launchBrowser();
try {
for (const doc of jobDocuments) {
const signDate = doc.type === 'clockOut' ? clockOutDate : clockInDate;
await stampAndRender({ doc, employeeID, jobID, signDate, browser, tx });
}
await tx.commit();
return { success: 1 };
} catch (error) {
await safeRollback(tx);
handleActivityError(error, 'Sign Job Documents');
}
}The 7:02 AM scenario resolves correctly and provably. The technician acted at 7:02. The request landed at 7:45 after the network recovered. The signed document reads 7:02. The 43-minute gap is fully visible in the workflow's request history and not a discrepancy in the labor record.
// Sample code for workflow query
const handle =
client.workflow.getHandle(`cico-${scheduleID}-${process.env.app_env ironment}`);
const { state, history } = await handle.query(QUERY_NAME); // history: [{ requestId, step, status, at }, …] — the full forensic record for this shift That's the difference between a system of record and a system you have to defend.
8. Hybrid cloud with zero-trust data security#
Field operations involve sensitive data such as personnel records, site locations, payroll figures due to which clients demand strict data sovereignty.
The solution uses Temporal Cloud as the orchestration layer while keeping all business data encrypted client-side via a Custom Data Converter (AES-256-GCM, keys never leave the on-premise network). Temporal Cloud stores and routes encrypted blobs without knowing their contents. The execution plane decrypts locally.
Temporal Cloud acts as a blind orchestrator managing the sequence of events and retries, but never seeing actual PII in plaintext. This delivered the benefits of a managed service (high availability, no cluster maintenance) while satisfying client data-sovereignty requirements.
This pattern applies broadly to regulated industries: adopting Temporal Cloud need not mean giving up data control.
9. Scalable by design: Graceful Saturation#
The system needed to handle thousands of simultaneous workers across hundreds of job sites, with a morning spike when everyone clocks in within the same 15-minute window.
Temporal's pull-based worker polling model provided a natural solution. Rather than thousands of simultaneous HTTP requests overwhelming the server, Temporal persists all signal events in its task queue the moment they arrive. A fixed-size worker cluster — just 5 Kubernetes nodes — pulls tasks as capacity allows, working through the backlog steadily.
No clock-in is lost or fails during the spike. The cluster runs near 100% utilization without falling over. Complex horizontal auto-scalers were avoided altogether. The team called it "Graceful Saturation": the system can saturate with work and remain stable, maximizing cost efficiency without operational fragility.
The daily work report: cron without the cron problems#
Every active job needs an end-of-day report that contains manpower counts, weather notes, site conditions, and safety observations all rendered to PDF and emailed to supervisors. Previously a person had to remember to do this, per job, every day.
We replaced it with a Temporal Schedule firing one workflow at 5:00 PM daily:
//Sample code for daily work report workflow
async function dailyJobReportWorkflow() {
const date = today();
await backfillMissingReports(date);
const { reports } = await renderReportPdfs(date);
const outcome = { total: reports.length, sent: 0, failed: 0 }; for (const report of reports) {
const r = await emailReport(report).catch(() => ({ success: 0 }));
r.success ? outcome.sent++ : outcome.failed++;
}
log.info('Daily work report run complete', outcome);
return outcome;
}The difference from a cron job is the difference between "it ran" and "it ran, and here is exactly which of the 40 jobs failed last Tuesday and why." Each activity retries independently so that one job's PDF failure doesn't sink the entire run and every execution leaves a durable, inspectable history.
The Control Tower: real-time visibility from Workflow History#
A side effect of moving to Temporal was an explosion of observability the team hadn't fully anticipated.
Because every action a field technician takes is a discrete event in a Temporal Workflow, that event stream became the data source for a real-time operations dashboard — internally called the "Control Tower":
Active sites & heatmaps. At a glance, executives see which sites have 15 active workflows and which have 2. Resources can be rebalanced during the day, not after.
Throughput and backlog. Live counters show tasks in progress versus completed. If "Tasks Completed" lags far behind "Tasks in Progress" by midday, a manager can investigate before it cascades into a schedule delay.
Blocker alerts. Technicians can send a "Blocked: Waiting on Materials" signal. The workflow records it and the dashboard highlights it instantly, marking it red on the board so that operations can dispatch what's needed before idle time compounds. Management shifted from reactive (finding out at day's end) to proactive (solving it in the moment).
Historical insights. Because Temporal retains a full history of every workflow, the team can query months of data: which crews consistently finish 20% faster, how many hours were lost to weather last quarter. The workflow history became a performance dataset available by design, not by instrumentation.
The team also integrated OpenTelemetry to trace requests end-to-end: from a tap on the iPad, through the API, into the Temporal workflow, and out to third-party systems like payroll. When something went wrong, they could identify not just that a workflow failed, but exactly why. Time-to-diagnosis on field-data issues dropped from most of an afternoon to minutes.
What we actually got#
We were deliberate about not putting everything on Temporal. Most of the platform is still ordinary request/response. We moved exactly the flows where the hard part was duration, coordination, or proof — and the returns concentrated in three places.
| Metric | Before | After |
|---|---|---|
| Payroll accuracy | ~80–85% | Above 99% |
| Tool losses per site | $50K+ | Near zero |
| Audit preparation | Weeks of paperwork | On-demand export |
| Time-to-diagnosis on field issues | Hours | Minutes |
Deleted complexity. Idempotency, dedupe, ordering, and retry classification used to be scattered defensive code accreted from individual incidents. They're now properties of the workflow's identity and message contract — written once, in one place, where an engineer can read the business rule.
Reliability as a strategy#
Field operations in construction and infrastructure are, at their core, a workflow problem. Every hour a crew spends on site is an event that needs to be captured, sequenced, validated, and paid. When that process is built on fragile tooling with apps that freeze in basements, GPS that fails underground, and databases that drop events during network interruptions, then the cost isn't measured in uptime metrics. It's measured in idle crews, compliance penalties, stolen tools, and disputes over timesheets that nobody can prove.
Temporal didn't just improve the reliability of individual systems in this stack. It changed the architectural posture of the entire platform — from a collection of fallible services trying to stay in sync, to a single durable execution that survives whatever any individual component does.
The human remains the star of the show. But Temporal is the director behind the scenes, ensuring that no matter what goes wrong with connectivity, servers, or infrastructure, every action that technician takes is recorded — exactly once, in the right order, with the right timestamp.
That's what Durable Execution looks like outside the glamorous use cases. And it turns out, that's where it matters most.
This post was written by the engineering team at Xgrid — a Certified Temporal Cloud Partner — based on production implementation work building large-scale field operations platforms. Learn more at xgrid.co.