Qualtrics: Simpler Cross Cloud Durable Execution

Abstract

When services need to communicate with other services, there are several options which might include APIs, queues such as Kafka / AWS SQS, etc. If latency must be low, communication via APIs probably makes sense. If the communication between services can be asynchronous and latency can be high, queues might make sense.

When communicating between services across different clouds, communication between services can be more difficult. Exposing APIs across the public internet is obviously more difficult than exposing APIs internally in a single cloud. The most common pattern for integrating services across clouds are public APIs but several issues can arise with APIs.

Cross cloud API communication requires: The API service must expose public (or somewhat public) api endpoints. These endpoints probably need to enable: Encryption Authentication / access rights Rate limiting (429s with too many calls) API clients must be able to access the public API (possible via IP allow-listing, security credentials etc) API clients must handle api failures by retrying.

About the Presenter

Sandesh is a Senior Software Engineer at Qualtrics where he has spent the past 6 years contributing to the Data Pipelines platform and developing internal tooling for NLP systems to simplify data enrichment tasks. He is an avid Gopher and can often be found engaging in local meetups talking about the Go programming language. Recently, he's been focussed on enriching text data with NLP and ML with a pipeline built on Temporal.