Orchestration and scheduling

A pipeline is several tasks with an order: extract orders and extract payments, then build staging, then run the tests, then build the marts, then refresh the dashboard. Drawn out, it's a directed acyclic graph (DAG): tasks are nodes, dependencies are arrows, and there are no loops.

An orchestrator runs the graph: it starts each task when the ones it depends on have succeeded, runs independent tasks in parallel, retries failures, records every run and alerts someone when things go wrong. Airflow is the best known; Dagster and Prefect are common alternatives, and many warehouses and dbt have simpler schedulers built in.

# a daily DAG in Airflow's TaskFlow style (names and imports vary between Airflow versions)
@dag(schedule="0 6 * * *", start_date=datetime(2024, 9, 1), catchup=False)
def shop_daily():
    orders = extract_orders()
    payments = extract_payments()
    staged = build_staging(orders, payments)
    tested = run_tests(staged)
    build_marts(tested)

Schedules are often written in cron syntax: five fields for minute, hour, day of month, month and day of week. 0 6 * * * means at minute 0 of hour 6, every day. */15 * * * * is every 15 minutes, and 0 2 * * 1 is 2:00 every Monday.

A scheduled run usually covers a period, its data interval, and runs after that period has ended: the run for 14 October starts early on the 15th, once the day's data is complete. Tasks should use the run's own interval ("process 14 October"), not today's date, which is what makes reruns and backfills work (next lesson).

Not everything should run on the clock. A sensor or event trigger starts the work when the data actually arrives (a file lands, an upstream job finishes), which is better than guessing a time and hoping.

Set retries with a delay for flaky steps, a timeout so a hung task doesn't block everything, and an alert that reaches a person, with an owner for each pipeline.