AI Driven, Reliable Medical Diagnostics#

Existing software for medical imaging devices forces doctors and diagnosticians to manually examine every chart, which is time consuming and error prone. Zebra Medical Vision (ZMV) provides an integrated solution, powered by machine learning to automatically identify medical conditions. The result is doctors spending more differentiated time with patients, providing them confidence that nothing critical was missed during diagnosis.

There is no room for error in ZMV’s system as the output will be directly used by doctors to make a diagnosis. To provide a consistent and safe analysis, ZMV is built on a complex data processing and machine learning pipeline which runs patient scans through a litany of proprietary algorithms. Any given step within this pipeline can be lengthy and may depend on dozens of external services.

How was this problem solved before?#

ZMV’s pipeline was originally a coordinated series of microservices written in Python. The pipeline receives files (medical scans) and processes them, eventually spitting a series of labels out into a well understood table format. The pipeline was developed organically, with individual teams implementing each microservice using the tooling and style of their choice.

While this worked great initially, it quickly led to serious disorganization. Data would enter the pipeline and would occasionally not emerge from the other end. Debugging high level issues with the pipeline was overwhelming, as there was no consistency in tooling or design from stage (microservice) to stage.

The challenge was a mix of Redis and Cron jobs and the pipeline required a lot of manual intervention. When it was failing, you had to massage it, check it every x times and move it from one stage to another manually. You didn’t have a clear picture of what happened to a single element and why. An element that had started the pipeline did not always make it through the end of the pipeline and the only option was to re-run this element through the pipeline again from the beginning. The data analysts became frustrated so they didn’t trust the pipeline and would have to manually check the data against the original source to feel confident.

It became clear that the pipeline architecture wasn’t just frustrating for ZMV developers; it was also affecting the business.

Cleaning up#

As the reliability and visibility of the pipeline started to degrade, they were already researching alternatives. Their experience helped them generate a wish list of capabilities for the V2:

• Out of the box visibility
• Can scale to 10’s of millions of data points
• Traceable
• Highly reliable
• Supports automatic retries
• Provides throttling controls
• Long running operations

At first they discussed using a more traditional solution like Airflow,but doubted it's ability to scale to meet their needs. Conductor was also a contender, but was ruled out immediately due to its configuration-driven nature.

In the end, it was a lot more convenient and intuitive to write the business logic in code with Temporal, instead of configuration. Temporal felt like the most natural solution.

Temporal migration#

Medical imaging requires some explicit knowledge to get right. When scans are done for a patient, multiple images are taken to ensure a comprehensive set of data is collected. This set of scans, taken together, is known as a series. Unfortunately, a series is not enough to represent imaging for a patient, as multiple series of different types (X-Ray, CT, etc.) may be needed to form a complete picture. The name for multiple series taken with a common goal is a study.

Before Temporal, teams across ZMV had no standard for how they processed medical images. Some services had been implemented to work with series, while others had chosen to work at the study granularity. The ZMV team decided that standardizing this API across the company would be a great benefit.

The problem was that many teams were already storing data in the series format. It wasn’t enough to change the API; the data also needed to be migrated. Orchestrating this migration seemed like a great first task for Temporal.

For each study, we needed to download all of its series, un-archive them, repackage them into one archive that represents the study, and upload them back through the pipeline. In addition, we discovered metadata inconsistencies inside the images, which we had to deal with during the migration. Using Temporal allowed them to discover and validation the business logic.

The result was not only a successful migration, but also immediate visibility about the state of the old pipeline and its data. The ZMV team was now confident about the state of data in the system and what had failed to make it through the pipeline.

Running the new pipeline allowed us to make sure that the studies we have are viable and safe to be used in our algorithms. We did find some errors in several scans (metadata inconsistency inside the scans).