Build a Reactive Data Streaming App with Python and Apache Kafka | Coding In Motion

https://cnfl.io/coding-in-motion-episode-2 | How do you get live notifications from a service that doesn’t support live notifications? Can you watch an online store for price drops? Or track comments on someone else’s YouTube video? Or at work, can you react to changes in another department’s system, when that system doesn’t have a notification API? How do you turn the question-and-answer model of the web into a live-streaming system?

In this episode of Coding in Motion we’re going to build a solution that brings some data to life. Join Kris Jenkins in another step-by-step build as he demonstrates how to turn a static data source—YouTube’s REST API—into a reactive system that:

► Uses Python to fetch and process data from a static web API
► Streams that data live, from Python into a Kafka topic
► Processes the incoming source data with ksqlDB, watching for important changes
► Then streams out live, custom notifications via Telegram

LEARN MORE
► GitHub source code: https://github.com/confluentinc/coding-in-motion/tree/main/watching-the-river-flow
► Build a Data Streaming App with TypeScript/JavaScript and Apache Kafka | Coding in Motion: https://www.confluent.io/coding-in-motion/sound-and-vision-data-streaming-app/?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.coding-in-motion-watching-the-river-flow_content.livestreams
► Coding in Motion Playlist: https://www.youtube.com/playlist?list=PLa7VYi0yPIH0oXBG_zpZHte86wISquelj
► Learn more with Kafka tutorials, resources, and guides: https://cnfl.io/confluent-developer-coding-in-motion-02
► Use CODING200 to get $200 of free Confluent Cloud usage: https://cnfl.io/try-cloud-coding-in-motion-02
► Promo code details: https://cnfl.io/promo-code-details-coding-in-motion-02
► Register for more: https://cnfl.io/register-coding-in-motion-02

TIMESTAMPS
00:00 Intro
00:27 What Are We Building?
01:24 Setting Up A Basic Python Program
02:57 Planning Our Approach
04:00 Fetching Data From Google ("So let's do that.")
07:28 Handling Paging With Python Generators
17:39 Fetching Specific Video Data
22:10 Setting Up A Kafka Cluster
24:26 Defining A Persistent Data Stream
26:03 Setting Up The Python Kafka Library
31:27 Serializing and Storing Our Data
35:02 Detecting Stream Changes With ksqlDB
39:59 Creating A Telegram Alert Bot
43:42 Setting Up An HTTP Sink Connector
46:58 Defining And Triggering The Alerts
50:59 Retrospective
53:02 Outro

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#streamprocessing #python #apachekafka #kafka #confluent