Using Apache Kafka as Cloud-Native Data System ft. Gwen Shapira

https://cnfl.io/podcast-episode-189 | What does cloud native mean, and what are some design considerations when implementing cloud-native data services? Gwen Shapira (Apache Kafka® Committer and Principal Engineer II, Confluent) addresses these questions in today’s episode. She shares her learnings by discussing a series of technical papers published by her team, which explains what they’ve done to expand Kafka’s cloud-native capabilities on Confluent Cloud.

Gwen leads the Cloud-Native Kafka team, which focuses on developing new features to evolve Kafka to its next stage as a fully managed cloud data platform. Turning Kafka into a self-service platform is not entirely straightforward, however, Kafka’s early day investment in elasticity, scalability, and multi-tenancy to run at a company-wide scale served as the North Star in taking Kafka to its next stage— a fully managed cloud service where users will just need to send us their workloads and everything else will magically work. Through examining modern cloud-native data services, such as Aurora, Amazon S3, Snowflake, Amazon DynamoDB, and BigQuery, there are seven capabilities that you can expect to see in modern cloud data systems, including:
1. Elasticity: Adapt to workload changes to scale up and down with a click or APIs—cloud-native Kafka omits the requirement to install REST Proxy for using Kafka APIs
2. Infinite scale: Kafka has the ability to elastic scale with a behind-the-scene process for capacity planning
3. Resiliency: Ensures high availability to minimize downtown and disaster recovery
4. Multi-tenancy: Cloud-native infrastructure needs to have isolations—data, namespaces, and 3. performance, which Kafka is designed to support
5. Pay per use: Pay for resources based on usage
6. Cost-effectiveness: Cloud deployment has notably lower costs than self-managed services, which also decreases adoption time
7. Global: Connect to Kafka from around the globe and consume data locally

Building around these key requirements, a fully managed Kafka as a service provides an enhanced user experience that is scalable and flexible with reduced infrastructure management costs. Based on their experience building cloud-native Kafka, Gwen and her team published a four-part thesis that shares insights on user expectations for modern cloud data services as well as technical implementation considerations to help you develop your own cloud-native data system.

EPISODE LINKS
► Cloud-Native Apache Kafka: https://cnfl.io/cloud-native-apache-kafka-episode-189
► Design Considerations for Cloud-Native Data Systems: https://cnfl.io/cloud-native-data-systems-design-considerations-episode-189
► Software Engineer, Cloud Native Kafka: https://jobs.lever.co/confluent/0b668a3b-0f25-4d23-b671-4e45300f1fed
► Join the Confluent Community: https://cnfl.io/join-community-episode-189
► Learn Apache Kafka on Confluent Developer: https://cnfl.io/confluent-developer-epiaode-189
► Demo: Event-Driven Microservices with Confluent: https://cnfl.io/event-driven-microservices-demo-episode-189
► Use PODCAST100 to get $100 of free Confluent Cloud usage: https://cnfl.io/confluent-cloud-podcast100-episode-189
► Promo code details: https://cnfl.io/podcst100-details-episode-189

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#cloudnative #serverless #apachekafka #kafka #confluent