Micro Free Course — 100% Free Learning
All lessons in this module are free to learn. Sign in with Google to save your progress.
Ingestion and Orchestration
Ingestion patterns (batch, CDC, streaming); managed tools (Fivetran, Airbyte, Meltano) and when to build vs buy; Apache Airflow deep dive; modern orchestrators (Dagster, Prefect); scheduling, backfills, and idempotency.
Module Content
Change Data Capture (CDC) — Real-Time Database Replication
Three ways data gets from source to warehouse. Each has different cost, complexity, and latency profile. FIND_VIDEO: search 'data ingestion patterns batch CDC streaming' — recommended channel: Confluent / Kahan Data Solutions. Aim for 10 min or under.
Quiz: Ingestion Mechanics — Change Data Capture
The decision tree for ingestion. Most pipelines use batch; CDC for relational sources; streaming reserved for narrow use cases.
Case 2 — Real-Time Ingestion: Postgres CDC to Cloud Warehouse
Deploy a production-grade Change Data Capture (CDC) pipeline streaming OLTP database updates to an analytical warehouse using Debezium, Kafka, and replication slots.
Managed Ingestion Tools (Fivetran, Airbyte, Meltano)
Three tools, three philosophies. Pick by your team size, budget, and willingness to operate infrastructure. FIND_VIDEO: search 'fivetran vs airbyte vs meltano comparison' — recommended channel: Airbyte / Seattle Data Guy. Aim for 10 min or under.
Quiz: Build vs Buy Ingestion
The build-vs-buy decision per source. Most pipelines mix managed connectors for SaaS and custom code for OLTP.
Apache Airflow — The Workhorse Orchestrator
The dominant orchestrator in 2026. Verbose, mature, omnipresent. Knowing it is non-negotiable for DE work. FIND_VIDEO: search 'apache airflow tutorial DAG basics' — recommended channel: Apache Airflow / Astronomer / Seattle Data Guy. Aim for 11 min or under.
Quiz: Airflow Core Concepts & DAG Authoring
Airflow's primitives, the DAG concept, and the patterns that separate good DAGs from messy ones.
Modern Orchestration with Dagster — Software-Defined Assets
Newer orchestrators address Airflow's pain points. Both are credible alternatives with different philosophies. FIND_VIDEO: search 'dagster vs prefect vs airflow comparison' — recommended channel: Dagster / Prefect / Kahan Data Solutions. Aim for 10 min or under.
Quiz: Modern Orchestration — Dagster & Prefect
Three orchestrators, three philosophies. Pick by team preferences, team size, and what you're orchestrating.
Airflow in Production — Scheduling, Backfills & Retries
Three concepts that separate amateur pipelines from production-grade ones. FIND_VIDEO: search 'data pipeline backfill idempotent design' — recommended channel: Seattle Data Guy / Kahan Data Solutions. Aim for 10 min or under.
Quiz: Pipeline Reliability — Idempotency & Backfills
The triad: schedule, backfill, idempotency. Get them right and pipelines self-heal. Get them wrong and every retry is a crisis.
Case 3 — Pipeline Reliability: Idempotent Airflow DAG Refactor
Refactor a brittle, non-idempotent production Airflow DAG that produces duplicate rows upon retry into a rock-solid, idempotent pipeline using TaskFlow API and atomic writes.