Evolving Streaming Applications

The reality of software systems is that the business needs they intend to service will change over time. So applications must be created that are able to evolve and follow these changing needs.

Welcome to the world where we are building high-volume distributed streaming applications using systems like Apache Flink, Spark and Kafka. Applications that are assumed to run 'forever' never go down.

So what happens if a business need changes? How can you make a streaming application that can evolve without breaking all the downstream applications that depend on it? Roll out the new producers first? Roll out the new consumers first? How do I avoid going down? But wait! Systems like Kafka persist records for weeks; so how do you handle the fact that there can be several different schemas in the Kafka topic at a certain point in time? Can you deploy a new application that reads both formats?

In this presentation Niels Basjes (Avro PMC) will go into the ways bol.com has chosen to handle these effects in a practical way. He will describe how the "Message" format and the schema evolution features of Apache Avro are used in conjuction with Apache Flink to make applications really 'evolvable'.

How do we make sure all applications are able to find the schema specifications, what can we do to ensure schemas stay 'evolvable,' and what were the pitfalls we ran into? Join us and find out.