Batch vs streaming in plain English

Someone says the dashboard must be “real time.” What they mean is: “I am tired of answering questions with yesterday’s export.” What they sometimes get is a multi-month streaming project, three new failure modes, and a chart that still cannot explain last quarter’s refund policy. Latency is a product requirement. It is not a moral virtue.

This is Part 2 of How data actually moves. Part 1 gave you the path: sources, land, transform, serve. This part answers a timing question on that path: batch versus streaming, in plain English, for analysts who inherit pipelines and need to know when waiting is fine.

What you’ll learn

What batch and streaming mean without broker jargon first
How to translate “real time” into a latency budget people can staff
When daily or hourly batch is the correct architecture, not a compromise
Micro-batch as the quiet middle ground most companies actually run
A worked decision table for common analyst use cases

Two ways data can travel (and a third everyone pretends is temporary)

Forget product names for a minute. Think about mail.

Batch is the mail truck. Letters pile up. At a scheduled time, the truck leaves. You get a full bag. Efficient. Not instant.
Streaming is the courier who runs each letter as it is written. Faster feedback. More coordination. More ways for one slow stop to matter.
Micro-batch is a truck that leaves every few minutes. It is batch with a short schedule. Many “streaming” slides in real companies are micro-batch wearing a hoodie.

Three clocks for the same table: batch mail sack, micro-batch frequent trucks, streaming courier. Pick the clock that matches the decision.

On the four-stage path from Part 1, batch versus streaming is mostly about how often sources feed land, and sometimes about how transforms run after that. Serve layers (dashboards, exports) can refresh on their own schedule even when land is continuous. That mismatch is why a “streaming pipeline” can still feed a dashboard that updates every fifteen minutes. The network was fast. The product was not designed to be continuous.

Rule of thumb: Ask for the maximum acceptable age of the number at decision time. If nobody can answer in minutes, hours, or a day, you do not have a streaming requirement. You have a vibe.

Batch in plain English

Batch processing collects a window of data, then processes that window as a unit. Classic examples: a nightly warehouse load, an hourly rollup of events, a weekly finance extract, a monthly close file. The defining features are a schedule (or a trigger like “file arrived”) and a bounded set of records for that run.

Why batch still dominates analytics:

Business clocks are batchy. Board packs, payroll, inventory counts, and cohort analyses often need complete days or periods, not the last second.
Reprocessing is easier to reason about. “Rerun March 12” is a clear operation. “Rerun the stream from offset X while consumers are live” is a specialty skill.
Cost and complexity stay lower for many volumes. You pay for a job window, not always-on processing for every event.
Quality checks fit naturally. Row counts, uniqueness, and reconciliation against source totals are straightforward at end of load.

Batch is not “old-fashioned.” Batch is how most decision-ready numbers are still made. If your metric definition needs a full day of refunds before it is honest, streaming the partial day into a board tile is not sophistication. It is a faster wrong answer.

What “late” means in batch

Batch has two different lateness ideas mixed together in meetings:

Schedule lag: the job is designed to finish by 7 a.m. with yesterday’s data. That lag is intentional.
Failure lag: the job should have finished by 7 a.m. and is still running at noon. That lag is an incident.

Analysts often feel both as “the data is late.” Your ticket should say which one. Intentional schedule lag is a product choice you can challenge with a latency budget. Failure lag is an operations problem for the land or transform owner.

Streaming in plain English

Streaming processes events continuously (or near continuously) as they arrive. Think payment authorizations, clickstream, IoT sensors, fraud scores, inventory reservations. The system does not wait for “end of day” to start work. It maintains ongoing state: counts, windows, joins against slowly changing dimensions, alerts.

Streaming earns its complexity when the decision cannot wait for the next batch and acting on fresher data changes an outcome. Fraud blocks, live inventory, operational alerts for outages, and in-product personalization sometimes clear that bar. “I want the KPI tile to feel modern” usually does not.

Streaming also changes failure modes. Instead of one big job failing, you get lagging consumers, poisoned events, out-of-order arrivals, and the philosophical joy of “exactly once” delivery semantics. You do not need to master those terms on day one. You do need to respect that continuous systems need continuous ownership. A nightly batch with one on-call rotation is a different staffing model than a 24/7 event path.

Event time versus processing time

This is the gotcha that burns analysts who first meet streams.

Event time is when the thing happened in the world (user clicked, payment captured).
Processing time is when your pipeline saw and processed the record.

A mobile app can offline-buffer events and upload them hours later. Your stream “today” may include yesterday’s event times. Batch jobs hit the same issue with late files, but streaming dashboards make the mismatch more visible because people expect “now” to mean “now in the world.” When you define metrics on streams, say which clock you use. The metrics series stress on definitions applies harder when time itself has two meanings.

Micro-batch: the compromise that runs the business

Many warehouses and orchestration setups run every 5, 15, or 60 minutes. That is still batch: bounded windows on a short schedule. Latency is “minutes,” not “milliseconds.” Complexity stays closer to batch. For a huge share of analytics and operational reporting, micro-batch is enough.

Examples where micro-batch often wins:

Support volume dashboards for team leads
Marketing spend pacing through the day
Warehouse pick rates for shift supervisors
Product funnels that do not drive automated user-facing actions

If a stakeholder says “real time” and accepts “within fifteen minutes,” write that down and stop the architecture spiral. You just saved a quarter.

Build a latency budget before you build a pipeline

A latency budget is a sentence:

Decision: ________________
Consumer: ________________
Max age of data at decision time: ______ minutes / hours / days
Cost of being wronger but faster: ________________
Cost of being righter but slower: ________________
Action taken on fresh data: automated / human / none yet

Fill it with real answers. “CEO might glance at it” is not a budget. “On-call pages if error rate exceeds 5% for 10 minutes” is a budget. “Finance closes books T+2” is a budget. “Sales manager plans calls each morning with yesterday’s pipeline” is a budget.

Notice the last line: action. Streaming without an action is entertainment infrastructure. If the only action is a human reading a chart in a weekly meeting, daily batch is usually correct.

Worked example: four stakeholders, four latencies

Same company, same order events. Different needs. This is how you stop one “platform real-time initiative” from eating every use case.

Use case	Decision timing	Max useful age	Mode	Why
Board monthly revenue	Monthly meeting	Days after month end	Batch (daily + close)	Needs complete refunds and accounting rules
Sales pipeline stand-up	Each morning	< 12 hours	Batch nightly or micro-batch	Human planning, not auto routing
Warehouse pack station screen	Continuous shift	< 2 minutes	Micro-batch or stream	Operators act continuously
Card fraud block	At authorization	< 1 second	Stream / online service	Automated decision in the product path

Latency budget chart showing board sales warehouse and fraud use cases from days down to sub-second needs

If engineering only has budget for one path this quarter, do not start with the board pack “going real time.” Start where delay changes outcomes in the product or operations path. Board packs can stay batch forever and still be excellent.

A tiny schedule sketch (batch / micro-batch)

Orchestrators (Part 4) often express batch as schedules and dependencies. Conceptually:

# conceptual only (not a full Airflow install)
schedule: "0 * * * *"          # hourly micro-batch
tasks:
  - extract_orders             # land
  - build_orders_clean         # transform
  - refresh_ops_dashboard      # serve
sla_minutes: 45                # fail the run if older than this

What that design promises consumers:

Clock time	Data through	Notes
10:00 job starts	~09:00 to 09:59 window	Window edges depend on source lag
10:20 typical finish	Same window certified	Within 45-minute SLA
10:50 still running	Stale serve layer	Page on-call, do not silent-fail

That table is more useful in a stakeholder meeting than a slide that says “event-driven architecture.”

When delay is fine (a longer list than people admit)

Delay is fine when:

The decision happens on a human cadence (daily, weekly, monthly).
Completeness matters more than speed (refunds, chargebacks, late adjustments).
The metric is comparative across full periods (week over week, cohort month 1).
You need heavy joins and rebuilds that are cheaper in batch SQL.
Regulatory or finance processes already lock numbers on a calendar.
Nobody is automated on the number yet, so freshness cannot change outcomes today.

Delay is not fine when:

An automated system blocks, routes, prices, or alerts based on the data.
Operators work continuously and stale screens create physical or customer harm.
Fraud, safety, or abuse signals decay in minutes.
You already promised a freshness SLA in a customer-facing analytics product.

Most internal analytics work sits in the first list. That is not a failure of ambition. It is a match between decision speed and data speed.

Talking to stakeholders without starting a platform war

Latency conversations go wrong when they become identity conversations (“we are a real-time company”). Keep them operational:

Replace “real time” with a number and a unit.
Ask what decision changes at that freshness.
Ask whether the action is automated or human.
Ask what completeness is sacrificed if you speed up.
Offer a trial: micro-batch for two weeks with an explicit success metric (fewer escalations, faster ops response), not a vibe check.

You will still lose some arguments to fashion. Document the budget you recommended anyway. When the streaming project stalls, the written budget is how the team returns to something shippable.

How this sits on the four-stage path

Streaming and batch are not alternate paths that skip stages. They change the rhythm of stages:

Sources: databases can be snapshotted (batch) or change-captured (stream-ish). Apps can emit events.
Land: files and tables in nightly drops, or topics and continuous append logs.
Transform: scheduled SQL models, or continuous processors, or both (stream for ops, batch for finance).
Serve: live APIs, or refreshed BI extracts, or both with clear labels.

Hybrid is normal. Fraud streams. Finance batches. Marketing micro-batches. Your job is to label which number comes from which rhythm so nobody compares a 2-second fraud metric to a T+1 revenue metric and declares “data inconsistency” as if the clocks were the same.

Quality still applies. Late data, duplicate events, and schema drift do not become rare because you bought a streaming logo. They become different shapes of the same problems covered in the data quality series. SQL skill still matters for batch marts (SQL series). Python still shows up in jobs and analysis notebooks (Python series). Learn more paths from the Learn hub rather than assuming every freshness problem needs a new pattern.

Common mistakes

Real time as a status symbol. Latency without an action is theater.
One latency for the whole company. Board and fraud do not share a budget.
Streaming the incomplete day into period metrics. Completeness beats speed for closed periods.
Ignoring event time. “Now” in the pipeline is not always “now” in the world.
Calling micro-batch streaming in exec slides. Then understaffing the always-on promise you implied.
No freshness SLA on serve. Consumers invent their own expectations and feel betrayed daily.
Debugging only the dashboard clock. The tile may refresh every minute while land is still nightly.
Skipping reprocessing design. If you cannot rebuild a bad hour, you do not have analytics. You have a firehose.

How to practice this week

Pick three numbers you use: one strategic, one operational, one if you have an automated or near-automated use.
Write a latency budget sentence for each (max age + action type).
Find the actual refresh: source extract time, transform run time, dashboard cache time. Note the slowest stage.
Label each number batch, micro-batch, or stream based on reality, not aspiration.
In the next “we need real time” conversation, ask only: “What decision changes if this is 15 minutes old versus 24 hours old?”
Optional: document one metric that must stay batch for completeness reasons (refunds, adjustments, late events).

Next in this series: warehouses versus lakes versus “just a database,” a decision tree for non-architects that complements storage reading without redoing it.

Quick recap

Batch moves bounded windows on a schedule; streaming processes ongoing events; micro-batch is short-schedule batch.
Latency is a budget tied to decisions and actions, not a brand value.
Most analytics can be late by design if completeness and human cadence dominate.
Streaming earns complexity when delay changes automated or continuous outcomes.
Event time and processing time are different clocks; say which one a metric uses.
Hybrid rhythms are normal; label them so comparisons stay honest.

Sources

Apache Flink documentation. “Event Time” and “Processing Time” (clear intro to time semantics). https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/
Apache Kafka documentation. “Introduction” (event log mental model for streaming systems). https://kafka.apache.org/intro
Apache Airflow documentation. “DAGs” and scheduling concepts. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
Google Cloud. “Batch and stream data processing” overview (vendor-hosted, useful contrast language). https://cloud.google.com/dataflow/docs/concepts/beam-programming-model
dbt Labs. “About dbt” (batch-oriented transform workflows most analytics teams still run). https://docs.getdbt.com/docs/introduction