Orchestration in one metaphor

A pipeline that works once is a script. A pipeline that works every day is a kitchen. Someone has to open on time, cook steps in order, redo the burnt tray, and tell the dining room when brunch is late. Orchestration is that kitchen management for data work. It is not the recipe (your SQL). It is not the pantry (your warehouse). It is the schedule, the dependencies, and the retries that keep breakfast from becoming a mystery meat buffet.

This is Part 4 of How data actually moves. You already have the path (sources → land → transform → serve), timing (batch versus streaming), and homes (app DB, warehouse, lake). This part is the orchestration metaphor: schedules, dependencies, and retries, with Airflow and dbt as examples you can recognize, not install guides you must complete before Monday.

What you’ll learn

• What orchestration owns versus what transform tools own
• Schedules, dependencies, retries, and SLAs in human language
• How Airflow-style DAGs and dbt-style model graphs fit together
• A worked morning kitchen run for daily revenue
• How analysts should read failures without becoming full-time SREs

The kitchen metaphor (keep it boring on purpose)

Map the ideas once:

Orchestration answers: when does work start, what must finish first, what happens on failure, and who gets paged? It does not replace good models. A perfect schedule running bad joins still serves bad brunch.

Rule of thumb: If your “pipeline” is a chain of calendar reminders and tribal knowledge, you do not have orchestration. You have hope with a timezone.

Three ideas that do most of the work

1. Schedules: when the kitchen opens

A schedule says the work should start at a cadence: nightly at 2 a.m., hourly, every Monday, or when a file lands. Schedules create expectations. “The dashboard is fresh by 8 a.m.” is a schedule plus an SLA, not a personality trait of the warehouse.

Analyst tip: learn the schedule of any number you publish. If leadership looks at 9 a.m. and the job finishes at 9:30 on good days, you do not have a metrics problem. You have a kitchen open-time problem.

2. Dependencies: do not plate before the sauce is done

Dependencies say task B cannot start until task A succeeds. Extract orders before clean orders. Clean orders before revenue mart. Revenue mart before BI extract. Without dependencies, jobs race. Racing jobs create “sometimes correct” tables, the worst kind, because they pass screenshots and fail audits.

Dependencies can be:

• Task dependencies inside one orchestrated graph (A then B then C)
• Dataset dependencies (wait until a table partition for yesterday exists)
• Cross-system dependencies (wait for a SaaS export, then load)

When a number is wrong “only on Mondays,” ask what Monday dependency is different: late upstream file, longer runtime, overlapping jobs, or a weekly task that never got edged into the graph.

3. Retries: the burnt tray policy

Retries say what to do when a step fails: try again, wait, alert, or stop the downstream line. Not every failure should retry forever. A bad password should not retry 50 times. A flaky network blip might retry three times with backoff. A failed data test should often block serve rather than push known-bad numbers to the dining room.

Analyst tip: distinguish infrastructure failure (timeouts, permissions) from data failure (null spike, duplicate keys). Retries fix many of the first. The second needs a human definition or a source fix.

Airflow-shaped thinking (example, not an install lab)

Apache Airflow popularized a mental model many teams still use even on other tools: the DAG (directed acyclic graph). In plain language: a flowchart of tasks that does not loop forever, with a schedule and clear upstream/downstream edges.

Core ideas you will hear:

• DAG: the whole kitchen plan for a pipeline
• Task: one step (extract, run model group, quality check, notify)
• Operator / sensor: how the step runs or waits (implementation detail)
• Run: one execution for a logical date (yesterday’s brunch service)
• SLA / timeout / retry policy: how late is late, how many redos

You do not need to author Python DAGs to benefit. You need to read the graph when something breaks: which task failed, what was blocked downstream, which logical date is incomplete. That is how you write tickets that say “extract_billing failed for 2026-03-12, marts not refreshed” instead of “dashboard broken.”

dbt-shaped thinking (transforms inside the pantry)

dbt is not primarily an orchestrator for the whole company. It is a transform framework: SQL models, tests, documentation, and a dependency graph among models. Many teams still schedule dbt with Airflow, cron, a cloud scheduler, or a vendor job runner. The split is healthy:

• dbt (or similar): what SQL runs, in what model order, with what tests
• Orchestrator: when dbt runs relative to extracts, reverse ETL, and BI caches

dbt’s model graph is still orchestration-adjacent. ref() edges mean “build customers before orders_enriched.” Tests mean “do not silently ship duplicates.” Docs mean “the dining room can read the menu.” Part 5 of this series goes deeper on dbt conceptually. Here, remember: transform graphs and platform schedules are partners.

Worked example: the 2 a.m. revenue service

Daily net revenue for leadership by 8 a.m. local. Kitchen plan:

Conceptual graph (YAML-like, not production config)

# conceptual orchestration sketch
pipeline: daily_revenue
schedule: "0 2 * * *"
timezone: America/New_York
sla_finish_by: "07:30"

tasks:
  - id: extract_invoices
    retries: 3
  - id: extract_refunds
    retries: 3
  - id: dbt_staging
    needs: [extract_invoices, extract_refunds]
  - id: dbt_tests_staging
    needs: [dbt_staging]
    on_fail: block_downstream
  - id: dbt_mart_revenue
    needs: [dbt_tests_staging]
  - id: bi_refresh
    needs: [dbt_mart_revenue]
  - id: notify_fresh
    needs: [bi_refresh]

What a successful run looks like as a status table an analyst can understand:

Now the failure story: dbt_tests_staging fails uniqueness on invoice_id. Downstream stays blocked. BI still shows March 11 with a stale banner if you built that honesty in. Leadership sees an old number with a warning instead of a quietly wrong new number. That is orchestration plus quality working together, the same spirit as the data quality series.

What analysts should do when the kitchen is on fire

You are not always the on-call engineer. You still need a response pattern:

1. Name the consumer impact. “Revenue tile stale for board prep at 8:30.”
2. Find the first failed task or the last successful logical date.
3. Classify: extract empty, transform error, test failure, BI refresh only.
4. Route to the stage owner from your Part 1 ownership card.
5. Communicate last-good data time to stakeholders in one sentence.
6. Do not “fix” a number in the BI tool unless you are explicitly serving a temporary manual process with a sunset date.

This is also where metrics discipline helps. If the definition is fuzzy, every outage turns into a debate about what the number should have been. Keep specs tight (metrics series) so outages stay operational.

Waiting done right: sensors without busy loops

Sometimes a task should not start on the clock alone. It should start when a file arrives, when an upstream partition is ready, or when a SaaS export finishes. Orchestrators call these waits sensors or external triggers. The kitchen metaphor still works: do not start the omelet until the eggs are delivered, but also do not stand in the walk-in forever without telling the dining room brunch is delayed.

Good waits have a timeout and an owner. Bad waits poll forever, fail silently, or start on pure clock time while the upstream is empty, which produces successful green jobs with zero rows. Pair waits with freshness checks so “ran” is not confused with “useful.”

For analysts, the practical move is to learn whether your number’s upstream is time-triggered or arrival-triggered. Monday blanks often come from a weekend file that never landed while the schedule still fired.

Idempotency: reheating without double-salting

A practical orchestration word: idempotent enough. If a task runs twice for the same logical date, you should not double-count revenue. Designs that append blindly and never replace partitions create duplicate brunch plates. Designs that rebuild a partition for the date safely can retry without shame.

Analysts feel non-idempotent bugs as “the number jumped after a rerun.” When you see that, ask whether the job deletes-and-reloads the date slice or only inserts. That one question saves days of ghost hunting.

How this sits with SQL, Python, and the rest of the path

Orchestration coordinates work written in many languages. SQL models in the warehouse (SQL series), Python utilities for awkward APIs (Python series), quality tests, and serve refreshes all become tasks on a rail. ETL versus ELT still describes transform timing relative to load; the orchestrator is what makes either pattern repeatable. For broader learning routes, use the Learn hub. Governance questions about who may rerun prod jobs pair with data governance thinking: least privilege applies to buttons that rebuild tables, not only to SELECT.

Common mistakes

• Cron soup without dependencies. Everything starts at 2:00 and races.
• Retrying data test failures forever. Bad data does not become good with patience.
• Alert fatigue. Page on every warning and soon nobody answers the real fire.
• Orchestrator as a code dumping ground. Business logic belongs in tested models when possible, not only in opaque task scripts.
• No last-good communication. Stakeholders invent rumors during outages.
• Success green, empty table. A job can “succeed” at doing nothing useful without row checks.
• One giant task. “Run everything” fails as a unit and teaches nothing.
• Treating dbt as the whole platform scheduler when extracts and BI still need coordination (or the reverse: scheduling only extracts and leaving transforms manual).

How to practice this week

1. Pick one dashboard you trust in the morning. Find its upstream job name or schedule doc.
2. List the dependency chain in five boxes or fewer: extract → transform → test → mart → serve.
3. Write the finish-by time consumers assume, even if nobody documented an SLA.
4. Ask what happens on test failure: block, warn, or ignore. If the answer is “ignore,” that is your improvement backlog.
5. Draft a one-sentence outage template: “As of TIME, METRIC is last good for DATE because TASK failed; owner TEAM.”
6. Optional: open any orchestrator UI you have access to and identify one DAG or job that feeds your work. Bookmark it.

Next in this series: what dbt is conceptually (SQL models, tests, docs) and why analysts should care even if they never own the scheduler.

Quick recap

• Orchestration is kitchen management: schedules, dependencies, retries, and alerts.
• Recipes (models) and pantries (storage) still matter; orchestration makes them repeatable.
• Airflow-style DAGs are flowcharts of tasks; dbt-style graphs are transform dependencies often scheduled by something else.
• Block serve on data test failures when honesty beats false freshness.
• Idempotent date slices make reruns safe.
• Analysts add value by reading graphs, classifying failures, and communicating last-good time.

Sources

• Apache Airflow documentation. “Core Concepts” overview. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html
• Apache Airflow documentation. “DAGs.” https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
• dbt Labs. “About dbt” introduction. https://docs.getdbt.com/docs/introduction
• dbt Labs. “About ref function” (model dependencies). https://docs.getdbt.com/reference/dbt-jinja-functions/ref
• dbt Labs. “Tests” (generic data tests as quality gates). https://docs.getdbt.com/docs/build/data-tests
• Google Cloud. “What is a data warehouse?” (serve-layer context for scheduled transforms). https://cloud.google.com/learn/what-is-a-data-warehouse

More definitions in Key Terms →

Merch on Etsy

Open store →

You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Y Hat Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Y Pred Mouse Pad View on Etsy → Naive Bayes Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Bayesian Mouse Pad View on Etsy →