DATA ENGINEERING ROADMAP смотреть онлайн

▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:35 - DE Fundamentals I: Python, DS & Algorithms
2:17 - DE Fundamentals ||: Git, Linux Commands, Shell Scripts
3:23 - Databases: SQL, Relational, Non-Relational, Data-Modelling
4:08 - Business Intelligence (Insights for stakeholders)
4:35 - Data Lakes & Warehouses
5:22 - Distributed Systems: Hadoop, Cluster Tech, Map Reduce
6:18 - Data Processing: Spark
7:07 - Orchestration: Airflow
7:54 - Backend Frameworks: APIs, Flask, FastApi, Django
8:29 - Automation & Deployment: Docker, Ansible, Terraform
9:12 - Frontend: Tableau, PowerBI, Streamlit, Plotly
10:21 - Machine Learning: Scikit Learn, Pytorch, Tensorflow, Linear Regression
10:50 - Outro

Roadmap Diagram Link: https://github.com/syalanuj/youtube/blob/main/data_engineering_roadmap/Data%20Engineering%20Roadmap.png


# Roadmap Details
## FUNDAMENTALS I
Focus on fundamentals such as Python, DS & Algorithms,
## FUNDAMENTALS II
Focus on fundamentals such as Linux Commands, Shell Scripting, Git, Networking Concepts

## Database
Dedicate a full sprint to Databases and SQL, as you will interact with them a lot. Focus on DB Fundamentals, SQL, Database Modelling, ACID Transactions, Relational, Non Relational Databases. And also play and experiment with data as you go along.

## Data Lakes & Warehouses
Understand fundamentals of Data Lakes and Warehouses, OLAP vs OLTP. S3, GCS, Big Query, Redshift, Clickhouse, Normalization vs Denormalization. Processing Big Data using SQL

## Distributed Systems
Modern frameworks achieve big data processing using Distributed Systems. Understand fundamentals of Hadoop, Map Reduce, HDFS. Cluster Tech such as EMR, Dataproc, Databricks

## Data Processing
Data processing can be considered Trunk of Data Engineering. A programming language such as Python would be core. Get accustomed to tools such as Pandas, SQL, Spark, Beam, Hadoop, etc

## Orchestration
Orchestrate your piplelines using tools, where you define the flow and schedule of your tasks. Learn in detail on how to use Airflow and create DAGS. Get a glimse of other tools such as Luigi, Jenkins

## Backend Frameworks
This part somewhat overlaps with a software engineer. Sometimes you need to server your models and functionality. Learn how to create APIs with frameworks like Flask, FastAPI, Django

## Automation & Deployments
This is also an important step, where you automate deployments. Learn containerization with Docker. CI/CD with Github Actions, Infrastructure as Code with Terraform, Ansible

## Frontend & Dashboarding
It is a good to have skills, to be able to showcase your data and results when in need. Learning dashboarding tools such as Tableau, PowerBI or open source python frameworks such as streamlit, dash, plotly is a good start

## Machine Learning?
You will be working with fellow Data Scientists and ML Engineers. It is good to understand the tools that they use.



FOLLOW ME ON
MEDIUM: https://medium.com/@syal.anuj
INSTAGRAM: https://www.instagram.com/anujsyal/
TWITTER: https://twitter.com/anuj_syal
GITHUB: https://github.com/syalanuj
WEBSITE: https://anujsyal.com


#dataengineering #roadmap #python #bigdata #cloud #sql #datawarehouse