Surge 2016 - Bruce Lowekamp - Seatbelt: A modular approach to reliable service-to-service...

Skype builds its new services as microservices on the Microsoft Azure platform. Our services are deployed and managed independently, and while there is a requirement that they survive failure of a deployment, the HA solution is determined by the service team. While we’ve realized benefits from microservice architecture, one of the challenges we’ve faced is managing service-to-service (S2S) communication. In particular, the range of service requirements and variability in deployment patterns creates complexities that have contributed to numerous incidents as S2S calls fail.

For our solution we built Seatbelt, a library for S2S communication based on modular building blocks of basic functionality. The underlying functionality of Seatbelt is implemented in blocks providing concepts such as Circuit Breaker, Retry, and Resource Pools. The library provides a standard factory that supports typical patterns with a few tuning options—a single call can create the set of blocks required for both real-time and latency-tolerant services. But unusual requirements can be met by working directly with the building blocks to define behavior required for the particular S2S relationship.

In this talk I will describe the experiences at Skype that led to designing and building seatbelt, the way it is used for the "90%" case, and unusual use cases that use the building blocks directly.