DEB2022 SYD05 Data replication patterns 101

Karen Davis
Principal Data Engineer @ ThoughtWorks

Talk:
Data replication patterns 101
At the core of data engineering is the need to move data from one source to another. To do this successfully there are multiple architectural decisions a data engineer has to make: Batch, micro batch or streaming? What data storage to use? How to solve for idempotency? It can be difficult to navigate these decisions without understanding the relationship and dependencies between data use cases, replication patterns and supporting data infrastructure.
In this talk we will outline some common data replication patterns and their associated use cases. We will describe the constraints of each of the patterns, break them down into their core components and describe how to apply them to real world problems. We will share principles you can apply when choosing data infrastructure and detail what to consider when solving for idempotency.

Bio:
Karen is a Principal Data Engineer at Thoughtworks. Her background is software engineering, data engineering and machine learning. While at Thoughtworks she has worked with numerous clients building large scale data platforms. She is passionate about all things data and is especially interested in data architecture and ways to apply software engineering best practices to data and machine learning.