Data Engineering Beginner Project - Implementation using Airflow, Python, S3, Snowflake (1/?)

In this series about providing value with Data Engineering, there will be two kinds of videos: the concept and the implementation.

For every couple of videos, a new iteration will introduce new concepts to the platform - with its respective implementation. With this, more robustness and new features will be added to the platform, as this data product becomes more and more valuable.

This first implementation video will create the base for the next iteration, using technologies such as:
- Apache Airflow;
- Python (requests, pandas, boto3, awswrangler);
- Docker (with Dockerfile customization);
- Snowflake.

#DataEngineering #DataPlatform #BusinessIntelligence #DataDrivenFuture #DataPipelines #DataArchitecture

---
CHAPTERS

00:00 Intro
01:04 Create S3 Bucket
01:49 Create Access Key + S3 full policy
04:23 Setup Airflow locally with Docker
08:11 Install requirements for ETL in Jupyter
09:53 Create ETL
17:13 Customize Airflow Dockerfile
20:53 Create External Table in Snowflake
24:13 Next steps

---
? REFERENCES

GitHub Repo (w/ commit): https://github.com/dataengexplors/data-engineering-beginner-project/tree/0abf83035bfad6c9fa3cf12a336e3bc507a542ec

Install Docker Desktop: https://docs.docker.com/engine/install/

Running Airflow in Docker: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

Python environment: https://docs.python.org/3/library/venv.htm #:~:text=A%20virtual%20environment%20is%20created,the%20virtual%20environment%20are%20available.

Building a new image with Docker: https://docs.docker.com/engine/reference/commandline/build/

Snowflake External Table with S3: https://docs.snowflake.com/en/user-guide/data-load-s3-compatible-storage

---
? LINKS

Follow me on Twitter: https://twitter.com/DataEngExplors
Follow me on GitHub: https://github.com/dataengexplors

---

ABOUT ME

I have dedicated my career to all aspects of data, encompassing algorithm development for enhanced recommendations, diverse data visualization creation, processing unstructured data, and designing efficient solutions across AWS, Azure, Google Cloud, and on-premises environments. Currently serving as a Data Engineer in a private company, I possess hands-on experience in solving technical challenges while also leading teams in building optimal data structures. My passion for data engineering motivates me to stay updated with the latest industry trends, so I can use my expertise to unlock the full potential of data and deliver impactful outcomes.


For business inquiries: [email protected]