Eli Lilly: From Monolith to Durable Microservices for Drug Discovery

Abstract

At Eli Lilly, the engineering team transformed a monolithic Django application into a distributed microservices architecture orchestrated by Temporal, reducing antibody sequence analysis time from days to hours and accelerating the drug discovery pipeline.

Next-Generation Sequencing analysis in pharmaceutical research requires processing large volumes of sequence data through bioinformatics tools that demand intensive disk I/O and CPU resources. This session will explore how the team applied Domain-Driven Design principles and leveraged Temporal to develop a durable software product capable of efficiently processing bioinformatics workflows at scale.

The approach utilizes Temporal's Python SDK worker implementations, optimizing both I/O- and CPU-bound tasks through strategic use of thread and process pools with asyncio's run_in_executor method. Guided by domain-driven design principles, this strategy enabled the decomposition of the monolithic Django application into microservices deployed on Kubernetes.

As a result, pipeline execution times were reduced from days to hours while maintaining deterministic execution. The system achieved both horizontal and vertical scaling by simplifying workers, migrating core business logic into microservices, and implementing producer-consumer patterns with S3 streaming for efficient data processing.

This implementation showcases patterns for modernizing scientific computing workflows, integrating legacy bioinformatics tools with cloud-native architectures, and providing a blueprint for similar transformations in pharmaceutical research pipelines.