How to load reference data to database with Python ETL Pipeline | Excel to Postgres

In this video we will build on from the previous session where we developed a Python ETL pipeline. Today we will see how to design an ETL Pipeline with Python to extract reference data from Excel files and load this data in PostgreSQL. We often have There comes a time when you need to incorporate the reference data sitting in flat files in your data warehouse to help translate a table or map data. Today, we will learn how to iterate over a directory, read multiple flat files, and load them into our postgres database with Python.

The goal is to read data from a network share and then load it in the database. So, the reference data is available for analytics and reporting.

Link to medium article on this topic: https://medium.com/@hnawaz100/python-etl-pipeline-to-load-reference-data-into-your-database-cc706e98cd58

Link to GitHub repo: https://github.com/hnawaz007/pythondataanalysis/blob/main/ETL%20Pipeline/etl_load_reference_data.py

Link to previous video: https://www.youtube.com/watch?v=dfouoh9QdUw

How to send emails with Python video: https://www.youtube.com/watch?v=cKsTNnVMKt8

Subscribe to our channel:
https://www.youtube.com/c/HaqNawaz

---------------------------------------------
Follow me on social media!

Github: https://github.com/hnawaz007
Instagram: https://www.instagram.com/bi_insights...
LinkedIn: https://www.linkedin.com/in/haq-nawaz/

---------------------------------------------

#ETL #Python #SQL

Topics covered in this video:
0:00 - Introduction to ETL
1:50 - The Setup
2:42 - Code ETL Pipeline
3:11 - Data Extract from Source Files
4:31 - Load data to PostgreSQL
6:11 - Test ETL Pipeline