📈 Introduction:

Are you ready to elevate your data engineering skills and create a standout portfolio project? This hands-on guide will walk you through building an end-to-end ETL data pipeline on Google Cloud Platform (GCP), using real-world weather data. By engaging with this project, you’ll not only learn but also apply your skills in a practical environment that mirrors what top companies are looking for in potential hires.

🦸 Who is this project for?

• Data Enthusiasts who are gearing up for their next role as a Data Engineer, Business Intelligence Engineer, Cloud Engineer, Python Developer, or GCP Developer.
• Experienced Data Professionals looking to deepen their expertise in Data Engineering and modern Cloud Technologies.

🎯 Learning Objectives:

By the end of this project, you'll be able to:

• Extract data from any public API using Python libraries like requests.
• Manipulate data efficiently with Pandas to transform raw data into actionable insights.
• Leverage GCP’s robust data warehousing tools like BigQuery to store and manage large datasets.
• Create dynamic data visualizations using Looker Studio to interpret and display your findings.
• Deploy and automate production-ready data pipelines using scalable Google Cloud services.

🏛️ Project Architecture:

The following diagram outlines how the project pieces fit together. It shows how we use tools like Cloud Functions to collect data from any API, Cloud Scheduler to automate the process, Cloud Storage, and BigQuery to store the data. It shows how everything connects, from starting with fetching API data to visualizing it in an easy-to-understand format with Looker Studio.

This diagram will help you visualize the steps and understand how each tool helps in building your data project.