NVIDIA: How to Manage GPUs in the Cloud and Keep Them Running, All the Time, Forever

Abstract

“Workflows” in Temporal are much more than a sequence of steps. In addition to traditional, DAG-based workflows, they may represent a Finite State Machine, an Actor, or any other process.

At NVIDIA, we operate a fleet of GPUs deployed across public and private clouds. Many developers use Temporal to deal with unreliable services. At NVIDIA we use it to manage unreliable hardware, networks, and configuration, and our solution involves all of the above Workflow types.

Not only does Temporal provide the building blocks for these various types of Workflows, they are interoperable via Signals, Child Workflows, Queries, and Updates. This means that a traditional (DAG) Workflow may represent an action taken by an Agent; or that a Signal to an Actor may form one step of a traditional Workflow; or that a State Machine Workflow may guard resources modified by another Workflow. All of these interaction patterns are used in our design.

The scale of Temporal allows us to represent each resource in our system as a Workflow, all the way from racks down to individual GPUs. Finally, because these fine-grained Workflows store little data and are usually idle, it is economical to run them on Temporal Cloud.

About the Presenter

I’ve been working with various versions of Temporal almost since its inception as Simple Workflow in Amazon’s Software Platform team. (In fact, Temporal Co-Founder & CTO Maxim Fateev was my mentor as a new-grad.) During my decade at Amazon, I wrote many applications based on Simple Workflow, before leaving for Google where I overlapped with Maxim again!

When Maxim founded Temporal and reached out, I was very excited to work on a technology that I believe in. I wrote the billing stack for Temporal Cloud, and later when I left to start my own consulting business it was to help others use Temporal more effectively.

Now I’m at NVIDIA where we’re developing a system to manage a very large fleet of GPUs, using Temporal of course!