menu

Workflow Orchestration

●10 ●89 ●146
calendar_today â€Ē schedule1 min read

Just completed Module 2 of the Data Engineering Zoomcamp 2026. Built production-ready data pipelines using Kestra to process 26 million NYC taxi trip records.
What I accomplished:

Orchestrated ETL workflows with Kestra
Ingested data from GitHub to GCS to BigQuery
Implemented partitioned tables for query optimization
Built MERGE operations for data deduplication
Automated monthly data loads with schedule triggers
Completed all homework questions (6/6)

Key learnings:

Template rendering with Pebble syntax
Handling trigger.date for manual vs scheduled runs
GCS storage class compatibility (REGIONAL vs STANDARD)
IANA timezone format for DST handling
BigQuery partitioning strategies

Cost efficiency:
Processed 2GB of data for less than one dollar, staying well within GCP's free tier.
Check out my project on GitHub: https://github.com/Derrick-Ryan-Giggs/module-2-workflow-orchestration
Thanks to DataTalksClub for this amazing free course.

1 Comment

â€Ē

Nice work, processing 26 million records for under a dollar is impressive and makes me want to try Kestra myself, how was the learning curve?

1 vote
ðŸ”Ĩ Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.
Join CoderLegion

More Posts

Tech Ecosystem Observatory: How I Built a Cloud-Native Data Pipeline to Track Global Tech Layoffs

Derrick Ryan - Mar 30

Sibling Rivalry? How to Make Kestra Tasks Talk to Each Other

Amara Graham - Apr 23

Batch Processing with Apache Spark

Derrick Ryan - Mar 7

From APIs to Warehouses: AI-Assisted Data Ingestion with dlt

Derrick Ryan - Mar 1

⚙ïļFrom 4 Hours to 3 Minutes: Automating GCP Ingress Provisioning with Ansible

ja.lourenco - Oct 27
chevron_left
Derrick Ryan Giggs
6.7k Points â€Ē 245 Badges
Kenya
71Posts
23Comments
6Connections
Aspiring Data Engineer | Learning Python, Java & Oracle Databases

On an exciting journey to become ... Show more

More From Derrick Ryan

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

Oracle GoldenGate 23ai: Powering Distributed AI with Real-Time Data Replication

Building the Sovereign Debt Observatory: An End-to-End ELT Pipeline on World Bank Debt Data for Low

Related Jobs

View all jobs →

Commenters (This Week)

2 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!