Dissecting GitHub Outage - Master failover failed

System Design for Beginners: https://arpitbhayani.me/sys-design
System Design for Experienced Engineers: https://arpitbhayani.me/masterclass
Become a member for exclusive in-depth videos: https://www.youtube.com/c/ArpitBhayani/join
Redis Internals: https://arpitbhayani.me/redis

Companies announce their planned maintenance, what happens during that? Could something go wrong while running maintenance?

GitHub team was switching their Master databases from one node to another; while doing this something went wrong and the new database crashed. This led to data divergence and a production incident that lasted over 5 hours.

In this video, we dissect this incident and understand what happens during planned maintenance, what went wrong with GitHub, how GitHub mitigated it, and understand some really cool things about switching databases and solving data divergence.

Outline:

00:00 Agenda
02:42 What happened?
03:29 Scaling reads with Read Replicas
04:40 Planned Database Maintenance
10:08 Database crashed and quick mitigation
11:44 Data Divergence between two masters
13:54 Remediating Data Divergence
18:23 Read Replica taking time to spin up

Check out the free course covering all GitHub outages → https://courses.arpitbhayani.me/github-outage-dissections/

# Arpit's System Design Masterclass

I teach a course on System Design where you'll learn how to intuitively design scalable systems. The course will help you

- become a better engineer
- ace your technical discussions
- get you acquainted with a massive spectrum of topics ranging from Storage Engines, High-throughput systems, to super-clever algorithms behind them.

I have compressed my ~10 years of work experience into this course, and aim to accelerate your engineering growth 100x. To date, the course is trusted by 500+ engineers from 9 different countries and here you can find what they say about the course.

Together, we will build some of the most amazing systems and dissect them to understand the intricate details. You can find the week-by-week curriculum and topics, benefits, testimonials, and other information here

https://arpitbhayani.me/masterclass.

### Other links

Newsletter: https://arpit.substack.com
LinkedIn: https://linkedin.com/in/arpitbhayani
Twitter: https://twitter.com/arpit_bhayani

You can also listen to Asli Engineering on the go on Spotify: https://open.spotify.com/show/7qMoamm2iZQrsPVm6IQLoD

You can find handwritten notes and text notes about this video on https://arpitbhayani.me.

### Things I use to make these videos

Apple iPad Pro 12.9 Inch: https://amzn.to/3jb4keI
Apple Pencil 2nd Generation: https://amzn.to/3DJ1Gq8
Boat Airdopes 621: https://amzn.to/3DIqGO6
GoodNotes Notetaking App: https://www.goodnotes.com/

Until next time, stay awesome :)

Yours truly,
Arpit
https://arpitbhayani.me
#AsliEngineering