Observability for pipelines

The pipeline was “fine.” The job was green. The dashboard opened. Then a VP asked why yesterday’s orders looked like a long weekend in the middle of the week. You discovered the load wrote zero new rows for six hours, a column type changed upstream, and the only alert went to a retired Slack channel. Green buttons lie when nobody defined red. Observability is how you notice that the path from source to serve is sick before leadership does.

This is Part 7 of How data actually moves, and the close of the series. Parts 1 through 6 covered the modern path, batch versus stream, storage choices, orchestration, dbt-style transforms, and environments. This part is the operations layer: freshness, volume, schema drift, tests, a red/yellow/green board, and who gets paged. Pair it with automated checks from Data quality, SQL you can trust (SQL series), and metrics that should not go dark silently (Metrics that matter). Full map: Learn.

What you’ll learn

What pipeline observability means for analysts (not only SREs)
Four signal families: freshness, volume, schema drift, and tests
How to use red, yellow, and green without alert fatigue
Who should be notified, and for what severity
A simple check board you can copy
Common mistakes, practice steps, and a full series recap of Parts 1 through 7

Observability is early news about the path

Observability for pipelines means you can answer, without archaeology: Did the data arrive on time? Is the amount of data plausible? Did the shape of the data change? Did our tests pass? Who owns the next action?

It is related to data quality, but not identical. Quality asks “is this dataset fit for use?” Observability asks “is the system that produces datasets behaving as expected right now?” You need both. A perfect historical cleanse does not help if today’s load never ran.

You do not need a six-figure platform on day one. You need a few honest signals on the tables people already fight about, a place to look (a board), and a human rota that is not “whoever notices first in Slack.”

Rule of thumb: If a broken pipeline can only be discovered by a surprised human reading a chart, you do not have observability. You have hope.

Four signals that catch most fires

Four pipeline observability signals: freshness, volume, schema drift, tests. Then red, yellow, or green. Not a vibe.

1. Freshness

Freshness answers: how old is the newest data we care about? Compare max event time or max load time to an SLA. “Orders fact should include data through yesterday by 7:00 a.m. local” is a freshness contract. “The job finished” is not the same thing if the job finished after writing nothing useful.

Warehouse platforms and schedulers often expose last modified times or job end times. Prefer a business-facing timestamp when you can: max order_ts in the mart, not only “dbt run completed.”

2. Volume

Volume answers: did we load a plausible amount? Absolute floors catch empty loads. Relative bands catch half files and double loads. Seasonality and weekdays matter; a Saturday drop may be normal. Tie volume checks to the same grain as the consumer table when possible.

3. Schema drift

Schema drift is when columns appear, disappear, rename, or change type upstream without your transforms noticing in time. A new nullability rule or a string where a number lived can zero out revenue after a silent cast. Detect drift by comparing information schema (or contract tests) to what your models expect. Fail loud on breaking changes; log non-breaking additions for review.

4. Tests

Tests are the assertions you already met in Part 5 and the quality series: unique keys, not nulls, accepted values, relationships, custom SQL sanity checks. Observability’s job is to surface test results on a board with history, not bury them in a CI log nobody opens until Friday.

Signal	Plain question	Example fail	Typical first response
Freshness	Is data recent enough?	Max order date is two days old at 9 a.m.	Check extract and orchestrator; delay dependent dashboards
Volume	Is row count plausible?	0 new rows; or 3× yesterday without promo	Inspect load files, filters, late duplicates
Schema drift	Did shape change?	`amount` became string; column dropped	Halt promote; fix model or push back on source
Tests	Do assertions hold?	Duplicate `order_id`; null revenue	Block mart consumers; patch logic or source

Red, yellow, green (without theater)

A simple severity model keeps humans sane:

Green: within SLA, tests pass, volume in band. No action required beyond normal watch.
Yellow: early warning. Freshness late but recoverable, volume slightly out of band, non-breaking schema addition, flaky test under investigation. Notify owners on business hours channels. Do not wake people at 2 a.m. for yellow unless the business is 24/7 on that metric.
Red: stop-the-line for dependent decisions. Empty critical mart, broken primary key, freshness past hard SLA for executive reporting, schema break that corrupts amounts. Page the on-call or the named owner immediately; consider banner on dashboards (“data delayed”).

Colors without thresholds are decoration. Write the threshold next to the light: “Red if max order_ts older than 26 hours on weekdays.”

Also decide what green does not mean. Green means “pipeline signals OK,” not “the business strategy is correct” and not “every metric definition is morally pure.” Keep semantic debates in metric reviews (metrics series), not in the ops board.

Who gets paged?

Paging is expensive. Use it for red on tables that matter.

Primary owner: the person or rotation who can restart jobs, read orchestrator logs, and open a fix PR.
Secondary / escalation: platform or engineering lead if primary misses the window.
Consumer lead (notify, not always page): finance or ops partner when dashboards must show a delay banner or meeting should slip.
Do not page: the entire company Slack; every analyst “for awareness”; the CEO for a yellow volume blip on a sandbox table.

Write a tiny RACI for the top five tables. If ownership is “data team,” that is not a name. Put a rotation calendar even if the rotation is two people trading weeks.

Analysts often sit adjacent to on-call rather than holding the pager. Still learn the board. When you see yellow, you can stop building a narrative on half data. When you see red, you can delay the deck instead of inventing a story. That is professional, not passive.

Worked example: a simple check board

Below is a morning board for a fictional commerce analytics stack. Three critical datasets, four signals, traffic-light status. This can live in a spreadsheet, a wiki table, a lightweight internal page, or a proper observability tool later. The design matters more than the vendor.

Pipeline check board with rows for orders mart, customers dim, and marketing spend, columns for freshness volume sche…

Dataset	Freshness SLA	Volume rule	Schema	Tests	Overall	Owner
`fct_orders_daily`	By 07:00, data through yesterday	New rows within 60-140% of 28-day same-weekday median	Contract match	unique order day key; amount not null	Red / Yellow / Green	DE rotation
`dim_customer`	By 07:30 daily	Row count not below prior day minus 5% without ticket	Contract match	unique customer_id	R/Y/G	DE rotation
`fct_marketing_spend`	By 09:00 (vendor lag)	Channels present ≥ baseline set	Contract match	spend >= 0	R/Y/G	Marketing analytics

Example SQL sketch for freshness and volume on the orders mart (run after the load, log the result):

WITH bounds AS (
  SELECT
    MAX(order_date) AS max_order_date,
    COUNT(*) AS rows_yesterday
  FROM analytics.fct_orders_daily
  WHERE order_date = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
),
baseline AS (
  SELECT APPROX_QUANTILES(day_rows, 100)[OFFSET(50)] AS median_same_weekday
  FROM (
    SELECT order_date, SUM(order_count) AS day_rows
    FROM analytics.fct_orders_daily
    WHERE order_date BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -56 DAY)
                        AND DATE_ADD(CURRENT_DATE(), INTERVAL -8 DAY)
      AND EXTRACT(DAYOFWEEK FROM order_date)
          = EXTRACT(DAYOFWEEK FROM DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY))
    GROUP BY 1
  )
)
SELECT
  max_order_date,
  rows_yesterday,
  median_same_weekday,
  CASE
    WHEN max_order_date < DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY) THEN 'RED_FRESHNESS'
    WHEN rows_yesterday = 0 THEN 'RED_VOLUME'
    WHEN rows_yesterday < 0.6 * median_same_weekday
      OR rows_yesterday > 1.4 * median_same_weekday THEN 'YELLOW_VOLUME'
    ELSE 'GREEN'
  END AS status
FROM bounds CROSS JOIN baseline;

Wire the status into your board or alerting tool. Tune bands after two weeks of false yellows. The first version will be slightly wrong; silence is worse than imperfect thresholds you improve.

Optional: emit lineage and run events to an open standard such as OpenLineage so multiple tools can share “what ran” and “what it produced.” Even without that stack, keep a run log: job name, start, end, rows written, status.

Lineage, incidents, and the human runbook

When red lights up, people need a short runbook:

Confirm the signal (not a flaky query against the wrong schema).
Check upstream: extract job, API quota, file drop, orchestrator queue (Parts 1 and 4).
Check transform: dbt or SQL job logs, failed tests (Part 5).
Check environment: did someone deploy to the wrong target (Part 6)?
Communicate: banner, Slack to consumer leads, ETA if known.
Fix, verify green, write a three-bullet incident note the same day.

Stewardship culture (ownership, incident habits, access) deepens in the next phase of AMS content. For today, observability without a runbook is just expensive anxiety.

Common mistakes

Only monitoring job success. Green job, empty table, sad VP.
Alerting on everything. Fatigue; then real red is ignored.
No owner on the board. Orphan lights help nobody.
Thresholds never tuned. Permanent yellow teaches people to look away.
Ignoring schema drift until BI breaks. By then the deck is already wrong.
Hiding status from analysts. Consumers keep publishing from bad data out of innocence.
No link to quality tests. Observability and validation should share the same critical tables.
Paging people for sandbox toys. Protect on-call focus for certified prod assets.

How to practice this week

Pick three tables that would embarrass you if wrong in a leadership meeting.
Write one freshness SLA and one volume rule for each in plain English.
Build a one-page board (Sheet is fine) with red/yellow/green and owners.
Run a manual freshness query three mornings in a row; note false alarms.
Confirm where test results already live (dbt, CI, scripts) and link them from the board.
Agree who is primary for one red scenario this month. Put the name on the calendar.

Quick recap

Observability gives early news: freshness, volume, schema drift, tests.
Red/yellow/green needs written thresholds and restraint on pages.
Owners and runbooks turn lights into action.
Start with a simple board on critical marts; tool up later if needed.
Quality series checks and pipeline signals reinforce each other.

Series recap: How data actually moves (G1 to G7)

This series was built for analysts who inherit pipelines and need a mental model, not a vendor certification. Here is the arc in one place.

G1. Sources, land, transform, serve. A modern path without vendor wars. Know which stage broke before you rewrite the dashboard.
G2. Batch vs streaming. When delay is fine and when it is not. Match latency to the decision, not to the hype.
G3. Warehouses, lakes, and “just a database.” A decision tree for non-architects so storage words stop being intimidation spells.
G4. Orchestration. Schedules, dependencies, retries. Something has to run the work in order and tell you when it failed.
G5. dbt conceptually. SQL models, tests, docs; staging, intermediate, marts. Shared transform logic with lineage you can follow.
G6. Environments. Dev, stage, prod; promotion paths; secrets; who writes trusted tables. Laptop truth is not company truth.
G7. Observability. Freshness, volume, schema drift, tests; boards; pages. Notice sickness before the screenshot does.

Together these parts answer: Where does data come from, how fast, where does it live, what runs it, how transforms are structured, how changes get promoted, and how you know the path is healthy.

Where to go next

Keep sharpening the skills that make pipeline literacy useful day to day:

Data quality for profiling, validation, and scorecards on the tables you just learned to watch.
SQL series for the queries behind models, checks, and marts.
Python for analytics when exploration, light automation, or charts sit beside the warehouse.
Metrics that matter so the tables you protect map to decisions, not vanity tiles.
Learn for the full AMS path map.

The next practice layer on this site’s roadmap is stewardship: ownership, catalogs, access, incidents, and working with Legal without panic. Pipelines move data. Stewards make sure the right humans care for it after it lands. You do not need a fancy title to start acting like one: name owners on your board, write grain on your marts, and refuse to ship a number you cannot rebuild.

If you only remember one thing from this series: data work is a path with stages, schedules, environments, and signals. When something looks wrong in a chart, walk the path before you invent a business story. That habit alone will save you years of elegant, confident mistakes.

Sources

Research and further reading used for this article:

dbt Labs: Source freshness (freshness as a first-class check on upstream data)
dbt Labs: Data tests (assertions that belong on the observability board)
OpenLineage documentation (open standard for job and dataset lineage events)
Great Expectations documentation (conceptual validation expectations and data docs mindset)
Snowflake: LAST_CHANGE_COMMIT_TIME and related freshness ideas (warehouse-side ways to reason about when data changed)
Google Cloud: BigQuery JOBS information schema (inspecting what ran and when for operational debugging)
Analytics Made Simple: Data quality (dimensions, validation, scorecards)
Analytics Made Simple: SQL series (query foundation for checks and marts)
Analytics Made Simple: Learn (series map and next paths)

Keep going

Same lessons in your feed

Short diagrams and hooks on Instagram, X, and Facebook.

@analyticsmadesimple Follow @AnalyticsSimply Follow Analytics Made Simple Follow