Airflow hello: your first DAG as a ticket rail

The ticket says “run the nightly load.” In practice that means: wait for the file drop, load staging, run transforms, ping the warehouse tests, and only then refresh the dashboard extract. On a good week one person remembers the order. On a bad week two jobs start at once, the transform reads half a file, and the dashboard looks confident while empty. Orchestration is how teams stop treating memory as infrastructure.

This is Part 1 of Airflow hello DAG. You will treat a first DAG as a ticket rail: a visible track of tasks with clear handoffs, not a dumping ground for all Python you own. We keep the install light and the ideas heavy. For broader pipeline context see data pipelines, for transform habits see the dbt lab and data quality, and for the full map open Learn.

What you’ll learn

What Apache Airflow is for (and what it is not)
DAG, task, operator, and schedule in plain English
Why “ticket rail” is a better mental model than “run my script”
A minimal first DAG structure you can read without being an Airflow expert
How dependencies turn a checklist into a graph
Common first-DAG mistakes and a practice plan

Airflow in one honest sentence

Apache Airflow is an open source platform to author, schedule, and monitor workflows as code. You define a DAG (directed acyclic graph): a set of tasks with dependencies and no cycles. Airflow’s scheduler decides when a DAG run should start. Workers (or task runners, depending on your executor) execute the tasks. The UI shows what is running, what failed, and what is waiting.

Airflow is not your warehouse. It is not dbt. It is not a replacement for good SQL. It does not magically make bad tasks reliable. It is an orchestrator: it orders work, records history, and gives humans a place to look when the night broke.

Rule of thumb: Put in Airflow the steps you would write on a runbook sticky note. Keep heavy business logic in the tools those steps call (SQL, dbt, Spark jobs, APIs), not inside giant Python operators if you can help it.

The ticket rail metaphor

Think of a physical ticket rail in a kitchen or a support desk: tickets move left to right through stations. You can see which station is stuck. You do not cook the entire meal inside the rail; the rail coordinates stations.

A good first DAG looks like that:

Station 1: Confirm the raw file or API extract arrived
Station 2: Load to staging
Station 3: Run transforms (for example dbt build)
Station 4: Run data tests or a row-count gate
Station 5: Notify or trigger a downstream refresh

Each station is a task. Edges between stations are dependencies. If station 2 fails, station 3 should not pretend success. That is the whole point of a rail instead of five unrelated cron lines.

Ticket rail DAG: check extract, load staging, run transforms, run tests, notify. Arrows show left-to-right dependenci…

Core terms without the fog

Term	Plain meaning	Kitchen analogy
DAG	Workflow definition (the menu and order of stations)	The full ticket path for one dish type
DAG run	One execution of that workflow for a logical date	Tonight’s tickets for that dish
Task	One unit of work in the DAG	One station on the rail
Operator	Template for how a task runs (Bash, Python, empty, etc.)	The type of station equipment
Schedule	When new DAG runs are created	When the kitchen opens a new batch
Dependency	Task B waits for task A	No plating before cooking
XCom	Small messages between tasks (use sparingly)	A short note pinned on the ticket

You will also hear executor (how tasks are actually run: local, Celery, Kubernetes, and so on). For a first mental model, ignore executor drama. Focus on clear tasks and honest dependencies. Part 2 covers retries and sensors; those sit on top of this rail.

Why “as code” matters

Airflow DAGs are Python files in a repo (or a managed equivalent). That means:

You can code review the runbook the same way you review a dbt model
History lives in git, not in a click-ops console someone inherited
Environments can promote the same DAG definition with different connections

The flip side: Python that builds the DAG runs in a special context. Heavy data work inside the DAG file at parse time can slow the whole scheduler. Keep DAG files thin: declare structure, call out to jobs. If you already check SQL carefully (AI SQL checks apply here too when tools draft operators), apply the same skepticism to generated DAG code.

A minimal first DAG (read-along)

Below is a teaching sketch, not a copy-paste production template. Names and imports vary slightly by Airflow version. The shape is what matters: default args, a DAG context, tasks, then dependencies.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    "owner": "analytics",
    "depends_on_past": False,
    "email_on_failure": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

with DAG(
    dag_id="hello_orders_rail",
    default_args=default_args,
    description="Ticket rail: extract check -> stage -> dbt -> test gate -> notify",
    schedule="0 6 * * *",  # 06:00 UTC daily; set timezone policy with your team
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["hello", "orders"],
) as dag:

    start = EmptyOperator(task_id="start")

    check_extract = BashOperator(
        task_id="check_extract",
        bash_command="test -f /data/incoming/orders_{{ ds }}.csv",
    )

    load_staging = BashOperator(
        task_id="load_staging",
        bash_command="python /opt/jobs/load_orders.py --date {{ ds }}",
    )

    run_dbt = BashOperator(
        task_id="run_dbt",
        bash_command="cd /opt/dbt && dbt build --select tag:orders",
    )

    quality_gate = BashOperator(
        task_id="quality_gate",
        bash_command="python /opt/jobs/check_orders_counts.py --date {{ ds }}",
    )

    notify = BashOperator(
        task_id="notify_success",
        bash_command="echo 'orders rail ok for {{ ds }}'",
    )

    end = EmptyOperator(task_id="end")

    start >> check_extract >> load_staging >> run_dbt >> quality_gate >> notify >> end

Read it as a ticket rail:

start / end bookend the run for readability in the graph view
check_extract fails fast if the file is missing (Part 2 will talk sensors for waiting)
load_staging and run_dbt do real work via scripts, not 200 lines inline
quality_gate is a deliberate station: transforms green does not mean counts make sense
{{ ds }} is the data interval date string Airflow injects so runs are parameterised by day

Dependencies: the rail edges

The line start >> check_extract >> ... sets a linear rail. Real life branches:

# After load, run two independent transforms, then join for tests
load_staging >> [run_dbt_orders, run_dbt_customers]
[run_dbt_orders, run_dbt_customers] >> quality_gate

Rules that keep rails sane:

No cycles. A cannot wait for B while B waits for A.
Fail closed. Downstream tasks should not run if upstream failed (default behaviour for normal deps).
Prefer explicit edges over hidden coupling through shared files with no task link.
Do not fake success. A notify task that always runs “green” after failures trains people to ignore the rail.

Schedule, start_date, and catchup (the confusing trio)

Three settings confuse every first user:

schedule (or legacy schedule_interval): how often new runs are created
start_date: the earliest logical date the DAG is allowed to consider
catchup: whether Airflow should create backfill runs from start_date up to now

For a hello DAG, set catchup=False so enabling the DAG does not fire a year of historical runs by surprise. When you do need history, run a deliberate backfill with eyes open. Align schedule timezone policy with your warehouse and product teams so “daily 6am” means the same morning for everyone.

Worked example: mapping a real ticket to tasks

Ticket from analytics ops:

Every morning we need yesterday’s orders in the mart before 8am local. File lands in S3 around 5:30. Load, dbt orders models, confirm row count within 5% of yesterday, then Slack #data-status.

Task breakdown:

Ticket phrase	Task id	What success means
File lands in S3	check_extract (or a sensor later)	Object exists for the logical date
Load	load_staging	Staging table replaced or appended correctly
dbt orders models	run_dbt	Selected models build without error
Row count within 5%	quality_gate	Gate script exits 0 only if rule holds
Slack #data-status	notify_success	Message sent only after gate passes

Example DAG run result strip: check_extract success, load_staging success, run_dbt success, quality_gate success, not…

That green strip is what stakeholders actually want: not “Airflow is installed,” but “yesterday’s orders cleared the rail.” When something fails, the strip shows which station broke so you do not restart everything from the kitchen door.

What to put in a task vs outside Airflow

In the task: invoke a job, pass the date, set timeouts, map exit codes to success/fail
Outside the task body when possible: multi-hundred-line transform logic, secret material, ad hoc analysis notebooks
In connections and variables: warehouse credentials, bucket names, environment-specific paths
Not in git: passwords and tokens; use a secrets backend or environment injection

This mirrors environment discipline from modern data teams: orchestration coordinates, systems of record still own the heavy lifting. dbt remains a great transform station on the rail; Airflow should call it, not reimplement it.

Observability for humans

On day one, agree on three operational basics:

Who owns the DAG? A real team name in owner and tags
Where do failures go? Slack, email, PagerDuty, or a ticket queue
What is the SLO? “Mart ready by 8am local” is better than “job usually finishes”

Name tasks after business stations, not after temporary scripts. Future you will thank present you at 2am.

Common mistakes

One giant task. A single Bash blob that does load + transform + notify hides the failure point.
Cron soup instead of deps. Five independent schedules that “usually” finish in order are not a rail.
catchup surprises. Enabling an old start_date with catchup true can stampede the warehouse.
Business logic only in operators. Unreviewable Python balloons; prefer callable jobs and dbt models.
Silent notify. Alerts on success only, or alerts that fire even when the gate failed.
No quality station. “dbt finished” is not the same as “numbers are plausible.”
Hard-coded dates. Always prefer the run’s logical date templates over yesterday’s constant.
Treating Airflow as a notebook host. Long interactive analytics does not belong on the scheduler path.

How to practice this week

Write a five-station rail on paper for one real daily job you already run by hand or cron.
Translate that paper rail into a DAG sketch with task ids and one line each for success criteria.
If you have a sandbox Airflow, implement a hello DAG that only uses EmptyOperator and Bash echo so you learn the UI without touching prod data.
Add a deliberate failing task and practice reading the graph and logs.
List which station should call dbt versus custom Python for your world; keep the boundary clean.

Part 2 of this series covers retries, sensors, and when Airflow is the wrong tool. Do not skip that if you are about to wait on flaky files or wrap every spreadsheet refresh in a DAG.

Quick recap

Airflow orchestrates workflows as code: DAGs, tasks, schedules, and visible history.
Treat a first DAG as a ticket rail of stations with honest dependencies.
Keep DAG files thin; put heavy logic in jobs and transform tools you already trust.
Learn schedule, start_date, and catchup before you enable anything near production.
Map real tickets to task ids and success criteria so the graph matches the business.
Next: retries, sensors, and when not to use Airflow at all.

Sources

Apache Airflow documentation (stable): https://airflow.apache.org/docs/apache-airflow/stable/index.html
Apache Airflow, “DAGs”: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
Apache Airflow, “Operators and hooks”: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html
Apache Software Foundation, Airflow project site: https://airflow.apache.org/
AMS, Learn map: https://analyticsmadesimple.com/learn/
AMS, Data pipelines series: https://analyticsmadesimple.com/series/data-pipelines/