Airflow hello: retries, sensors, and when not to use it

The file is late. Your DAG starts on schedule, the first task fails, Slack lights up, someone hits “clear and rerun,” and by the third morning the team has taught itself a new job title: human sensor. Retries and sensors exist so people stop playing that role. Used badly, they also hide real outages under a blanket of “it will work next try.”

This is Part 2 of Airflow hello DAG. Part 1 built the ticket rail mental model and a first DAG shape. This part covers retries, sensors, and the harder skill: knowing when Airflow is the wrong tool. Keep pipeline context from data pipelines, quality gates from data quality, and the wider map on Learn.

What you’ll learn

How retries differ from “run it again and hope”
When sensors help (and when they burn workers)
Timeout, poke interval, and mode choices in plain English
A worked late-file pattern that fails loudly with a deadline
Clear signals for not using Airflow
Common failure modes and a practice drill

Retries: recover from flukes, not from design bugs

A retry re-executes a failed task a limited number of times after a delay. Good retries assume the failure might be transient: a blip in the network, a brief warehouse queue, a rate limit that clears.

Bad retries assume the universe will invent a missing file or fix a wrong join if you wait five minutes. It will not. You will only delay the alert and stack overlapping work.

default_args = {
    "owner": "analytics",
    "retries": 2,
    "retry_delay": timedelta(minutes=10),
    # optional: retry_exponential_backoff=True in many setups
}

Guidelines that keep retries honest:

Low counts for loaders and APIs that can flake (1 to 3 is common)
Zero or one for pure logic bugs you expect to fail the same way every time
Idempotent tasks so a second try does not double-insert rows
Alert after final failure, not after every intermediate attempt, unless you truly want noise

If a task fails three days in a row and always succeeds on retry number two, you do not have a resilient system. You have a hidden dependency on timing. Fix the dependency (often with a sensor or an upstream SLA) instead of raising retries forever.

Retry rule: Retries buy time for the world to settle. They are not a substitute for a correct wait condition or a correct join.

Sensors: waiting as a first-class station

A sensor is a task that waits for a condition: a file in object storage, a partition in a hive table, an upstream DAG success, a row appearing in a control table. Sensors turn “human checks S3” into a rail station with a timeout.

Filled Airflow wait path: schedule, sensor, retries, alert

Classic example: do not start the load until yesterday’s extract object exists.

from airflow.sensors.filesystem import FileSensor

wait_for_orders = FileSensor(
    task_id="wait_for_orders_file",
    filepath="/data/incoming/orders_{{ ds }}.csv",
    poke_interval=60,   # seconds between checks
    timeout=60 * 60 * 2,  # fail after 2 hours
    mode="reschedule",  # free the worker between pokes when supported
)

Key knobs:

Knob	Meaning	Practical tip
poke_interval	How often to re-check the condition	Too aggressive hammers storage; too slow delays the rail
timeout	Max wait before the sensor fails	Align to the business SLO, not hope
mode=poke	Holds a worker slot while waiting	Fine for short waits; dangerous at scale
mode=reschedule	Releases the worker between checks	Prefer for long waits when available
soft_fail	Skip downstream instead of failing hard	Use rarely; easy to hide missing data

Sensors are not free. Hundreds of long poke mode sensors can starve real work. Prefer reschedule for multi-hour waits, or event-driven patterns (messages, deferrable operators in newer Airflow) when your platform supports them. Read the official sensor docs for the version you run; APIs evolve, the wait-vs-work idea does not.

Retries vs sensors vs SLAs

People mix these three:

Sensor: wait until a precondition is true, then proceed
Retry: after a task fails, try the same task again
SLA / deadline thinking: if success is late, page a human even if the task eventually finishes

Late file pattern that works in many teams:

Sensor waits up to a defined timeout for the extract
Load and transform use a small retry for transient warehouse errors
If the sensor times out, fail loudly; do not load partial inventiveness
Optional: a separate monitoring check that pages if the mart is still empty at 8am local

That last bullet matters. A DAG can still be “running fine” while the business deadline is already dead. Orchestration success is not the same as product success.

Worked example: orders rail with a late file

Business rule: mart must be ready by 08:00 local. Extract usually lands by 05:30. Sometimes vendor lag hits 07:00. After 07:30 waiting is pointless; finance will use the fallback process.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.filesystem import FileSensor

default_args = {
    "owner": "analytics",
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

with DAG(
    dag_id="orders_rail_with_sensor",
    default_args=default_args,
    schedule="0 5 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["orders", "sensor"],
) as dag:

    wait_for_file = FileSensor(
        task_id="wait_for_orders_file",
        filepath="/data/incoming/orders_{{ ds }}.csv",
        poke_interval=120,
        timeout=60 * 90,  # 90 minutes after the DAG starts
        mode="reschedule",
    )

    load_staging = BashOperator(
        task_id="load_staging",
        bash_command="python /opt/jobs/load_orders.py --date {{ ds }}",
        retries=2,  # warehouse blips only
    )

    run_dbt = BashOperator(
        task_id="run_dbt",
        bash_command="cd /opt/dbt && dbt build --select tag:orders",
        retries=1,
    )

    quality_gate = BashOperator(
        task_id="quality_gate",
        bash_command="python /opt/jobs/check_orders_counts.py --date {{ ds }}",
        retries=0,  # a bad count is not transient
    )

    notify = BashOperator(
        task_id="notify_success",
        bash_command="echo 'orders ready {{ ds }}'",
    )

    wait_for_file >> load_staging >> run_dbt >> quality_gate >> notify

Notice the intentional split:

Sensor owns lateness of the vendor file
Load gets more retries than the quality gate
Quality gate fails closed; bad counts should not “retry into truth”

Result panel: sensor timed out at 90 minutes for missing orders file, downstream tasks skipped, alert sent; contrast …

That timeout panel is a feature. A clear red “file never arrived” is cheaper than a green load of emptiness that poisons dashboards until lunch.

Sensor anti-patterns

Infinite patience. No timeout trains the vendor that late is fine and burns capacity.
Soft fail everything. Downstream “skips” look healthy in the UI while marts go stale.
Poke mode for multi-hour waits across dozens of DAGs without capacity planning.
Sensing the wrong thing. File exists but is zero bytes or yesterday’s copy; pair sensors with size checks or content gates.
Sensor as business logic. Complex rules belong in a small script or query, not a tangle of nested sensors.

When not to use Airflow

Airflow shines at multi-step, multi-system batch workflows with dependencies, history, and human operators. It is a poor default for every automation itch.

Situation	Prefer	Why not Airflow first
One SQL transform on a schedule	dbt Cloud schedule, warehouse tasks, or a single orchestrated job	DAG overhead for one station
True streaming / sub-minute events	Stream processors, queues, CDC tools	Airflow is batch-oriented at heart
Ad hoc analyst notebooks	Notebooks, scheduled notebook services carefully scoped	Not a substitute for exploration UX
Simple app cron inside one service	The app’s own scheduler or platform cron	Cross-system orchestration not needed
CI for code tests	GitHub Actions / GitLab CI	Different lifecycle and secrets model
You have no operators on call	Managed simpler tools until ownership exists	Airflow without owners becomes a museum of failed tasks

Also pause if the real problem is definitional. If two teams disagree on revenue grain, a more sophisticated DAG will only deliver the wrong number faster. Fix contracts and tests first (the dbt lab’s PR review habits transfer). Orchestration cannot invent semantic agreement.

Lightweight alternatives (know the menu)

Depending on stack, teams also use Prefect, Dagster, cloud-native schedulers (GCP Cloud Composer is Airflow-managed; AWS Step Functions and Glue workflows; Azure Data Factory), or vendor schedules inside ELT tools. The decision criteria stay similar: dependencies, observability, team skill, and whether you need a general graph or a narrow managed job.

For many analytics teams the winning combo is boring: managed extract tool + dbt for transforms + a thin orchestrator (Airflow or vendor) for cross-tool rails + warehouse tests. Do not adopt Airflow to feel enterprise. Adopt it when the ticket rail is real and recurring.

Operational checklist for retries and sensors

Document for each task: transient vs permanent failure modes
Set retries only where a second try can succeed without damage
Make loads idempotent or merge-safe before raising retry counts
Give every sensor a timeout aligned to a business deadline
Prefer reschedule or deferrable waits for long conditions
Alert on sensor timeout as a vendor or upstream incident, not as “Airflow is broken”
Review sensor count and mode in capacity planning, not only in DAG PRs

Common mistakes

Retries on bad data. Wrong logic fails the same way; fix the model.
No timeout on waits. Tasks that wait “until forever” hide outages.
Human sensors. If someone must click rerun daily, encode the wait or escalate the vendor.
Alert fatigue. Retry storms and soft fails train teams to mute channels.
Airflow for one SQL file. Over-tooling creates owners of platform pain without value.
Ignoring idempotency. Double loads after retry create duplicate facts and “unique” test fires at 9am.
Skipping quality stations because the sensor already “proved” the file exists.
Confusing green tasks with on-time data. Track business SLOs separately when needed.

How to practice this week

Pick one flaky job and classify last month’s failures: transient, late upstream, or logic bug.
For each class, decide: retry, sensor, or code fix. Write it in the DAG PR description habit from the dbt lab.
Add or tighten one sensor timeout to match a real stakeholder deadline.
Remove one unjustified retry from a task that always fails for the same reason.
List two automations on your team that should not move to Airflow and say why out loud.

If AI tools draft sensors and retry blocks for you, still verify timeouts, modes, and idempotency by hand. Generated orchestration code fails the same way generated SQL does: confident, plausible, and occasionally disastrous. The verification mindset in how to check AI-written SQL applies to DAG reviews too.

Quick recap

Retries help transient failures on idempotent tasks; they do not fix wrong logic or missing files.
Sensors encode waiting with poke intervals, timeouts, and modes; prefer capacity-friendly waits.
Separate late upstream (sensor) from flaky execution (retry) from bad counts (fail closed).
Align timeouts to business deadlines; green after 11am may still be a product miss.
Skip Airflow when you need streaming, single-station schedules, CI, or you lack owners.
Together with Part 1, you have a hello path: rail shape first, then resilient waiting and honest limits.

Sources

Apache Airflow, “Sensors”: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/sensors.html
Apache Airflow, “DAGs” (scheduling and task behaviour): https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
Apache Airflow, “Deferrable Operators & Triggers” (for modern async waits): https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html
Apache Airflow project site: https://airflow.apache.org/
AMS, Learn map: https://analyticsmadesimple.com/learn/
AMS, Data pipelines series: https://analyticsmadesimple.com/series/data-pipelines/