Sources, land, transform, serve

You open a dashboard on Monday and the number is wrong. Not slightly off, not “seasonality,” wrong. You ask the person who built the report last year. They left. You open the warehouse, find six tables with almost the same name, and three scheduled jobs that all claim to be the source of truth. Someone says “check the pipeline,” and you nod as if that sentence has a single meaning.

This is Part 1 of How data actually moves, a series for analysts who inherit pipelines, debug broken numbers, and need a mental model before the vendor slides start. We are not picking a cloud brand. We are naming the path every serious analytics stack walks, even when the tools change: sources → land → transform → serve.

What you’ll learn

• The four-stage path data takes from production systems to a chart you can defend
• What “landing” and “transform” mean when you are not a full-time data engineer
• Where ETL and ELT fit as loading styles, not as religions
• How to map ownership so a broken number has a next human, not only a next ticket
• A worked weekly revenue example you can reuse when you reverse-engineer a job

Why analysts need a path, not a product list

Most “data engineering” content starts with tools. Fivetran, Airbyte, dbt, Spark, Kafka, Snowflake, BigQuery, the list never ends. Tools matter. Starting with tools is how you memorize logos and still cannot explain why yesterday’s orders table is empty.

As an analyst, you do not need to design the entire platform. You do need to answer four questions when a number looks weird:

1. Where did this data originate?
2. Where was it stored first after extraction?
3. What cleaning, joins, and business rules changed it?
4. What object is the dashboard (or export, or model) actually reading?

Those four questions are the four stages. If you can walk them, you can partner with engineers, write better tickets, and stop treating the warehouse as a magic box that “should just be right.”

Rule of thumb: If you cannot name the stage where a number went wrong, you are debugging feelings, not a pipeline.

The four stages: sources, land, transform, serve

Here is the path as a single line. Real stacks add branches, retries, and exception tables. The line still holds.

1. Sources: where truth is born messy

Sources are the systems of record for some real-world event. Orders live in a checkout database or an ecommerce platform. Support tickets live in a helpdesk. Product usage lives in an events store or SaaS analytics tool. Finance truths may live in an ERP. Marketing spends live in ad platforms. People data lives in HRIS. None of these systems were built to make your weekly board pack easy.

That is not a complaint. Transactional systems optimize for writing one order quickly, charging a card, or assigning a ticket. They optimize for current state, not for historical analysis. When you pull from a source, you inherit its grain, its timestamps, its soft deletes, and its “we fixed that field last March” history.

Analyst habit for sources: write down the system of record and the event or entity you care about. “Revenue” is not a source. Stripe charges, Shopify orders, or the ERP invoice table might be. If two sources both claim revenue, you have a definition problem before you have a pipeline problem. That is the same discipline you use when you define metrics in the metrics series: name the thing before you count it.

2. Land: copy first, argue later

Landing is the first durable home of data after it leaves the source. Teams call this a raw zone, a bronze layer, an ingest schema, a staging bucket, or “the dump tables.” The name matters less than the rule: land data as close to the source shape as you can afford, with enough metadata to reprocess later.

Why land at all? Because sources change, rate-limit, and forget. Landing gives you a frozen copy for a point in time. If finance changes a status mapping next quarter, you can rebuild transforms from landed history instead of replaying every API call from 2019.

Landing is not the same as cleaning. In a healthy path, raw tables still look a little ugly on purpose. Ugly raw data is honest. Pretty raw data often means someone already applied business logic you cannot see. When your dashboard is wrong, you want a place to ask: “Did the source send this, or did we invent it?”

Where that land zone lives depends on architecture. Object storage and lakes are common for files and events. Warehouse raw schemas are common for relational extracts. We will compare warehouses, lakes, and application databases in Part 3. For Part 1, remember the job: durable copy, source-faithful, reprocessable.

3. Transform: turn copies into business language

Transform is where analytics work becomes legible. You join customers to orders. You filter test accounts. You map status codes to “won / lost / open.” You convert time zones. You compute revenue recognition rules. You dedupe identities. You build the tables a human can query without reading the source schema documentation at 11 p.m.

Transforms may be SQL models, Python jobs, stored procedures, spreadsheet macros that somehow became production, or a mix. Modern teams often keep transforms as versioned code in the warehouse (dbt-style models are one popular pattern). Older stacks transform before load. That loading style difference is the ETL versus ELT conversation you have already seen on this site: extract and load first, then transform in the warehouse, versus transform before or during load. If you need a refresher on that tradeoff as a concept, start from the Learn hub and the existing ETL and storage posts rather than treating this series as a redo.

What matters for analysts is ownership of business logic. “Is a trial conversion counted on signup day or first paid invoice day?” is not a Spark configuration. It is a product and finance decision encoded as a transform. When numbers disagree across teams, half the time the extract is fine and the transforms encode two different answers.

Quality work lives here too. Null checks, uniqueness tests, accepted values, and volume assertions catch drift early. That is the bridge to the data quality series: a pipeline that moves fast without checks is a rumor delivery system with better uptime.

4. Serve: what consumers actually touch

Serve is the layer people believe is “the data.” BI datasets, semantic layers, reverse-ETL syncs into the CRM, exports to finance, feature tables for models, CSV dumps for a partner, even the spreadsheet that is still the monthly close. Serving is packaging: stable names, documented grain, access controls, and a refresh promise someone can trust.

A common failure mode: the transform layer is solid, but five dashboards each re-implement filters in the BI tool. Suddenly “active customers” has four definitions again. Serving well means the definition lives once, upstream of the chart, and the chart is mostly presentation.

Another failure mode: nobody documents which served table is certified. Analysts find a convenient table named orders_v2_final and ship a board slide. Six months later engineering drops that table because “nothing depended on it.” Serving is also a social contract: if it is public, it needs a steward.

How the path shows up in job titles and tickets

You will not always own all four stages. You still need to know who does.

When a dashboard is wrong, write the ticket against a stage. “Dashboard filter wrong” is serve. “Join drops half of orders after March” is transform. “Raw table empty since Saturday” is land or source. “Source system changed status codes” is source, with a transform update after. Vague tickets bounce. Staged tickets get fixed.

Worked example: weekly revenue for a small subscription product

Imagine you inherit a “Weekly Revenue” tile. Leadership trusts it until it disagrees with finance. Your job is not to rebuild the company stack. Your job is to walk the path.

Map the stages on paper first

Already you have places to look. You do not start by rewriting SQL randomly. You start by asking whether the tile’s number matches mart.revenue_daily, whether the mart matches raw, and whether raw matches a sample pull from the billing UI.

A tiny pseudo-SQL sketch of the transform

The real model may be fifty lines with tests. The idea often looks like this:

-- conceptual transform (not a full production model)
SELECT
  CAST(invoice_ts AS DATE) AS revenue_date,
  customer_id,
  SUM(amount_usd) AS gross_amount,
  SUM(refund_usd) AS refund_amount,
  SUM(amount_usd) - SUM(refund_usd) AS net_revenue
FROM raw.billing_invoices
WHERE is_test_account = FALSE
  AND status IN ('paid', 'partially_refunded')
GROUP BY 1, 2;

What that kind of model is trying to produce (toy numbers):

If the tile says $180 for March 1 but the mart says $180 and finance says $160, you are not in a “dashboard formatting” problem. You are in a definition problem: finance may exclude refunds posted later, or use invoice issue date instead of payment timestamp. The path did not fail silently. The path encoded one business rule, and another team used a second rule.

Ownership card for the same pipeline

Write this once and pin it where the team works:

Notice how the ownership card prevents the worst meeting: everyone staring at the tile and blaming “data” as if data were a person.

ETL, ELT, and other acronyms without the holy war

You will hear ETL (extract, transform, load) and ELT (extract, load, transform). In the four-stage path:

• Extract always starts at sources.
• Load into land always happens somewhere.
• Transform either happens before load (classic ETL tooling outside the warehouse) or after load (ELT-style transforms in the warehouse or lakehouse).
• Serve still sits after transforms, whatever loading order you chose.

Cloud warehouses made ELT common because they scale SQL well and keep intermediate tables cheap enough to store. That does not make ETL “wrong.” Some regulated flows still transform and validate before data is allowed into a shared store. Some event paths transform in stream processors before anything lands. Your job as an analyst is to ask where business logic runs and whether you can re-run it from raw, not to win a vendor debate.

If your org is still choosing storage shapes (warehouse versus lake versus “just Postgres”), treat that as related reading under storage options and Part 3 of this series. If you write SQL for a living, the SQL series is the language layer on top of whatever lands and transforms. If you clean and reshape extracts in Python, the Python series covers the analyst-side craft that often plugs holes while the formal pipeline catches up.

A 15-minute reverse-engineering checklist

When you inherit a number and have almost no docs, run this quick path audit before you rewrite anything:

1. Open the dashboard or export and write the exact metric title and filters you see.
2. Identify the serve object (dataset, view, extract). Note its name.
3. Find the transform that builds that object (model file, stored procedure, notebook job). Note grain in one sentence.
4. Find the land or raw inputs. Confirm last load time.
5. Name the source system in business language (billing, CRM, app DB), not only a schema name.
6. Write one open question per stage you could not fill. Those questions become tickets or coffee chats.

Fifteen minutes of path mapping often saves fifteen hours of guessing joins. It also shows respect for the people who built the stack under deadline pressure. Most inherited messes were reasonable once. Your job is to make the current path legible again.

What “good enough” looks like for a mortal stack

You do not need a magazine architecture to be useful. You need:

• Known sources with named owners
• A land zone you can query or at least inspect
• Transforms in versioned code (not only a BI calculated field museum)
• A short list of served objects that are certified for decisions
• Basic checks: freshness, row volume, uniqueness on keys
• A place where definition changes are written down

That list is deliberately boring. Boring pipelines make exciting analysis possible. Exciting, undocumented pipelines make exciting outages.

Common mistakes

• Treating the dashboard as the source of truth. The dashboard is a serve surface. Truth lives upstream or it does not exist.
• Cleaning only in the BI layer. Every chart reinvents filters. Definitions drift by Tuesday.
• Pretty raw tables. If land already applies heavy business logic, you cannot rebuild when rules change.
• No grain statement. “One row is one invoice line on payment day” prevents half of join disasters. Same idea as tidy data discipline elsewhere on this site.
• Silent dual pipelines. Marketing has one revenue path, finance has another, and both call themselves official.
• Ticket language without a stage. “Numbers wrong” is not actionable. “Land job missed 3/12 extract” is.
• Assuming more tools fix ownership. A new orchestrator without owners is a faster way to ship confusion.
• Skipping quality until after a public mistake. Tests are cheaper than apologies in the all-hands.

How to practice this week

1. Pick one number you personally report or use in a weekly meeting.
2. Write a one-page path: source system, land object, transform object, serve object. Leave blanks if you must. Blanks are findings.
3. For each stage, write one human owner (name or team), not a tool name.
4. Compare the served number to the transform table for the same day. Note any filter that exists only in the BI tool.
5. Open one raw or land table and confirm you can explain three columns without guessing.
6. Optional: draft a definition sentence for the metric and link it to the transform, the way you would in a metrics spec.

Next in this series: batch versus streaming in plain English, including when waiting until tomorrow is the correct architecture. For broader learning paths, use the Learn hub. When governance language shows up (who may access what, what “official” means), the data governance key term page is a useful companion, not a substitute for a working path map.

Quick recap

• Data moves through sources, land, transform, and serve, even when tool names change.
• Landing preserves source-faithful copies so you can rebuild later.
• Transforms encode business language and should be versioned and testable.
• Serve is what people touch; it should not re-implement core definitions.
• ETL and ELT are loading orders inside the same path, not opposing religions.
• Stage-aware tickets and ownership cards turn “data is wrong” into work someone can finish.

Sources

• dbt Labs. “About dbt” and docs overview (models, tests, and analytics engineering concepts). https://docs.getdbt.com/docs/introduction
• Apache Airflow documentation. “DagRun” and core concepts (how scheduled work relates to pipelines). https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html
• Google Cloud. “What is a data warehouse?” (vendor-hosted intro useful as a neutral mental model). https://cloud.google.com/learn/what-is-a-data-warehouse
• Snowflake. “Key concepts” overview (warehouse-oriented analytics concepts, example not endorsement). https://docs.snowflake.com/en/user-guide/intro-key-concepts
• Kimball Group. “Data warehouse bus architecture” and dimensional modeling primers (classic serve-layer thinking). https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/kimball-data-warehouse-bus-architecture/

More definitions in Key Terms →

Merch on Etsy

Open store →

You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Dashboard to Excel Ceramic Coffee Mug View on Etsy → Boxplot Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Box Plot, Whisker Plot View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy →