What is Data Engineering? Overview of the Data Engineering role and responsibilities | Python | SQL

In this video we will discuss the Data Engineering role. We will discuss what is data engineering and we will go over the roles and responsibilities of this roles and whether it might a good fit for you.

Data engineering is part of the big data ecosystem and it is closely linked to data analytics and data science. Data engineers work in the background yet they are critical to both data analytics and data science. The roles and responsibilities of a data engineer might vary depending on a company’s level of data maturity and skillset; however, there are some tasks, such as the extracting, loading, and transforming the data, that are common across the industry and to the role of a data engineer. If you want to deep dive into the theoretical part of data engineering then I’d recommend picking up a good book on ETL (extract, transform and load).

Data engineering involves the movement of data from one system (usually referred to as the source system) to another system target. The data engineers query data from a source (this is the extract part), then they perform some modifications to the data (this is the transformation phase), and then they store that data in a location (this is the load part), usually a database where it is available for querying.

#etl #dataengineering #dataengineer

Learn Python: https://www.learnpython.org/
Learn SQL: https://www.codecademy.com/learn/learn-sql

Link to GitHub repo: https://github.com/hnawaz007/pythondataanalysis

Subscribe to our channel:
https://www.youtube.com/c/HaqNawaz

---------------------------------------------
Follow me on social media!

GitHub: https://github.com/hnawaz007
Instagram: https://www.instagram.com/bi_insights...
LinkedIn: https://www.linkedin.com/in/haq-nawaz/

---------------------------------------------

Topics covered in this video:
0:00 - Introduction to Data Engineering
1:59 - Example of Data Engineering Role
3:16 - What Data Engineers do?
3:40 - Importance of Data Engineering
4:16 - Transformations Examples
5:13 - Data Pipeline
5:44 - Requirements for Data Engineering Role
7:26 - Data Pipeline Orchestration