📈 Introduction:

Are you ready to unlock the potential of Google Cloud Platform (GCP) and create a standout data engineering project? This hands-on guide will walk you through building an end-to-end ETL data pipeline on GCP using real-world weather data. By engaging with this project, you’ll not only learn but also apply your skills in a practical environment that mirrors what top companies are looking for in potential hires.

🎯 Project Definition:

Develop an ETL Data Pipeline to extract hourly weather information for a selected location. Deploy this data pipeline on Google Cloud Platform and set it to run automatically on an hourly schedule. Store the retrieved data in BigQuery. Aim to use Google Cloud services that minimize costs.

🦸 Who is this project for?

• Data Enthusiasts who are gearing up for their next role as a Data Engineer, Business Intelligence Engineer, Cloud Engineer, Python Developer, or GCP Developer.
• Experienced Data Professionals looking to deepen their expertise in Data Engineering and modern Cloud Technologies.

🎯 Learning Objectives:

By the end of this project, you'll be able to:

• Extract data from any public API using Python libraries like requests.
• Manipulate data efficiently with Pandas to transform raw data into actionable insights.
• Leverage GCP’s robust data warehousing tools like BigQuery to store and manage large datasets.
• Deploy and automate production-ready data pipelines using scalable Google Cloud services.

🏛️ Project Architecture:

The following diagram outlines how the project pieces fit together. It shows how we use tools like Cloud Functions to collect data from any API, Cloud Scheduler to automate the process, Cloud Storage, and BigQuery to store the data. It shows how everything connects, from starting with fetching API data to visualizing it in an easy-to-understand format with Looker Studio.