Handling a data incident

It is 9:14 a.m. A VP posts a screenshot in Slack: yesterday’s revenue on the executive dashboard is 18% higher than Finance’s close number. Three people already forwarded the chart. Someone asks if the campaign “worked.” Someone else asks if the warehouse is “lying.” Nobody is laughing. You own the domain that feeds that tile. Congrats. You just inherited a data incident.

This is Part 4 of Data stewardship at work. Parts 1 through 3 covered roles, finding a source of truth, and access. This part is operational: what you do in the first 24 hours when a number people trust is wrong (or might be wrong). We stay on the practice layer. For program language, see Data governance. For the pipelines and checks that reduce how often this happens, pair this with How data actually moves, Data quality, and Metrics that matter. Map of paths: Learn / Tutorials.

What you’ll learn

• What counts as a data incident for stewards (not only security breaches)
• A first-24-hours timeline you can run under pressure
• How to triage severity without theater or panic
• A red / yellow / green check board for wrong-number cases
• An incident log template and a worked Finance-versus-dashboard example
• Communication patterns that reduce rumor while you investigate
• Common mistakes and a Monday-morning practice drill

What is a data incident (for people like you)?

In security teams, “incident” often means unauthorized access, malware, or a leak. Keep that meaning. As a domain steward, you also care about a quieter class of problems: trusted outputs that are wrong enough to drive a bad decision.

For this series, treat something as a data incident when all three are true:

• Material: the number, list, or export is used for decisions, money, customers, or compliance, not a throwaway sandbox chart.
• Trusted: people already treat it as “the answer,” or it is labeled as the official metric / dashboard / export.
• Suspect or confirmed wrong: you have evidence of mismatch, missing load, broken definition, bad join, or similar, not only a vague “feels high.”

A wrong dashboard number is a data incident. A leaked customer file is also a data incident, and it usually escalates to Security and Legal immediately. This post focuses on the wrong-number path because stewards hit it weekly and often handle the first hours alone. The same discipline (contain, log, communicate, hand off) applies when the issue is privacy-sensitive, only with earlier escalation.

Rule of thumb: If people are already making decisions on it, treat a material mismatch as an incident until you prove it is not. Hope is not a severity level.

The first 24 hours: a calm timeline

You do not need a war room and a branded “SEV-1” channel for every mismatch. You do need a clock and a sequence. The goal of day one is not a perfect root cause essay. The goal is stop further harm, establish facts, communicate status, and leave a trail.

Hour 0 to 1: contain and label

Contain means limit how many people act on the bad number while you investigate. Options scale with impact:

• Add a clear banner or note on the dashboard: “Under review. Do not use for decisions until cleared.”
• If you control the publish path, pause the scheduled refresh or pin the last known-good snapshot (with a date stamp).
• Reply in the thread where the screenshot lives: “Acknowledged. Investigating. Do not redistribute until we post an update.”
• If the export is already in email, you cannot unsend it. You can still send a short correction notice once you know what failed.

Label means give the incident a short name and an owner. Example: INC-2026-03-12-rev-dash-v-finance, owner: you (or the on-call steward). Open a single log (ticket, doc, or issue). From this moment, facts go in the log, not only in chat scrollback.

Hour 1 to 4: triage severity and freeze the “what”

Triage answers three questions, not twenty:

• Who is affected? Executives only, one team, external partners, customers?
• How bad if we wait? Money moved, public statement, payroll, regulatory report, or “annoying internal chart”?
• How wide is the blast radius? One tile, one mart, one source system, or the whole warehouse?

Write a one-sentence problem statement you can defend later:

“As of 2026-03-12 09:14 local, the Executive Revenue tile for yesterday shows $1.18M while Finance close shows $1.00M for the same calendar day. Difference is about 18%. Tile is marked under review.”

Notice what is missing: blame, guesses about marketing, and “the BI tool is broken.” Facts first.

Hour 4 to 12: run the check board

Wrong numbers almost always fall into a short list of causes. Run them in order so you do not jump to “redefine the metric” when the load simply never finished. The check board below is the practice version of pipeline observability from the pipelines series: freshness, volume, schema, tests, then definition and grain.

Hour 12 to 24: communicate, fix or escalate, hand off

By end of day one you should have either:

• A confirmed cause and a temporary safe path (corrected number, banner removed with explanation, or explicit “use Finance source until mart is back”), or
• A clear escalation with evidence packet for engineering, platform, or vendor ownership, plus a status message for consumers.

Root-cause depth can continue after 24 hours. Consumer trust does not wait for a perfect postmortem.

Severity without theater

Use three levels so humans share a language. Adjust names to match your company if Security already has SEV numbers. The point is shared meaning, not jargon cosplay.

If you are unsure between yellow and red, ask: “Would I be okay if this number hit an external partner email tonight?” If no, treat as red until proven otherwise.

The wrong-number check board

Work top to bottom. Stop when you have a confirmed cause strong enough to act. Revisit lower rows only if the first fix does not close the gap.

Rows 1 to 4 are pipeline and quality territory. Rows 5 to 7 are metric and stewardship territory. Do not rewrite the metric on row 6 until rows 1 to 5 are green or explained. That habit alone prevents “definition churn” during every scare.

Incident log template (copy this)

Keep one log per incident. Short fields beat a novel. You can paste this into a ticket description or a shared doc.

# Data incident log

id: INC-YYYY-MM-DD-short-name
opened_at_local: 2026-03-12 09:14
opened_by: jane.doe
severity: yellow   # red | yellow | green-watch
status: investigating   # investigating | contained | resolved | false-alarm

## Problem (one sentence)
# As of ..., system A shows X while system B shows Y for ...

## Consumers / blast radius
# e.g. Exec revenue tile, weekly sales email, partner portal export

## Containment actions (with times)
# 09:20 banner on tile
# 09:25 reply in #exec-metrics: under review

## Timeline (facts only)
# 09:14 VP screenshot in Slack
# 09:18 steward confirmed 18% gap for calendar day yesterday
# 10:05 freshness check: max order_ts is T-2 (expected T-1 by 07:00)

## Checks run (link queries or paste results)
# freshness: FAIL
# volume: PASS for T-2; FAIL for T-1 (0 new rows)
# schema: PASS
# keys: not run yet
# metric def: not run yet

## Current hypothesis
# Overnight orders extract failed after 02:00; mart still shows prior day labeled as "yesterday"

## Decision for consumers (until next update)
# Do not use Exec tile. Use Finance close export for decisions.

## Next update by
# 2026-03-12 15:00 local

## Owners
# steward: jane.doe
# pipeline: data-platform on-call
# metric owner: finance.ops

## Close notes (fill when resolved)
# root cause:
# fix:
# prevention:
# who was notified of resolution:

The log is not theater. It is how you avoid three people “fixing” three different theories at once, and how you write a useful close note without reconstructing Slack archaeology.

Worked example: Finance close vs executive tile

Setup (toy numbers, labeled as example):

Hour 0 to 1: Jane puts a banner on the tile, posts in the thread, opens INC-2026-03-12-rev-dash-v-finance, severity yellow (internal exec decisions; not external filing yet).

Hour 1 to 4: She freezes the problem statement (gap about 18% for calendar yesterday). She does not yet say “marketing double counted.”

Check board run (abbreviated queries she might run in the warehouse):

-- Freshness: latest order event in the mart used by the tile
SELECT
  MAX(order_ts) AS max_order_ts,
  MAX(loaded_at) AS max_loaded_at,
  COUNT(*) AS row_count_yest
FROM analytics.fct_orders_daily
WHERE order_date = CURRENT_DATE - INTERVAL '1' DAY;

-- Expected by 07:00 local: max_order_ts through end of yesterday
-- Example result: max_order_ts is two days ago; row_count_yest = 0

-- Volume band: compare to same weekday last 4 weeks
SELECT
  order_date,
  COUNT(*) AS orders,
  SUM(net_amount) AS net_revenue
FROM analytics.fct_orders_daily
WHERE order_date >= CURRENT_DATE - INTERVAL '35' DAY
GROUP BY 1
ORDER BY 1;

Results in the example: the mart has zero rows for “yesterday,” but the tile still shows $1.18M. That is a second smell. Digging into the BI definition, the tile was set to “last non-empty day” after a past outage, so it quietly redisplayed the prior day under the label “yesterday.”

Cause chain:

1. Overnight extract failed (pipeline row red).
2. Mart stayed empty for the calendar day (volume / freshness red).
3. BI fallback reused prior day without relabeling (consumer presentation red).
4. Finance close was correct for calendar yesterday (source of truth for close decisions).

Consumer decision for the rest of the day: use Finance close. Fix path: restore extract, recompute mart, remove “last non-empty day” fallback or label it explicitly as “latest available day: DATE.” Prevention: freshness test fails the tile publish, not only the job green light. That last sentence is pure stewardship: you care about the decision surface, not only the job status light.

How to communicate without feeding the rumor mill

Under pressure, people fill silence with stories. You do not need a press release. You need short, timed updates.

First reply (minutes)

“Acknowledged the mismatch between [A] and [B] for [period]. Investigating. Please do not redistribute the chart. Next update by [time]. Owner: [name].”

Midday update

“Still investigating. What we know: [fact]. What we ruled out: [fact]. Safe source for decisions until further notice: [system]. Next update by [time].”

Resolution or handoff

“Root cause (summary): [one or two sentences]. Impact window: [dates/times]. Fix: [what changed]. Prevention: [check or process]. Tile status: cleared / still blocked. Full log: [link].”

Do not name-and-shame in the public channel. Do not over-promise “never again.” Do not invent precision (“exactly 17.84% caused by…” ) if you only know the class of failure.

When this is also a security or privacy incident

Escalate early if any of these appear: unauthorized access, export of personal data to the wrong party, credentials in a screenshot, a vendor asking for a full prod dump “to debug,” or a request to hide the issue from Legal. Wrong numbers and breaches can overlap (for example, a broken filter that exposed another tenant’s rows). In that case, containment includes access review, not only a dashboard banner. Part 6 covers calm collaboration with Legal and Security. Part 3 covered least privilege so fewer people can create those blasts.

Common mistakes

How to practice this week

1. Pick your top three decision dashboards. Write the safe alternate source for each (even if it is “Finance export” or “CRM report”).
2. Create a blank incident log template in your team wiki. Fill one with a past scare as a dry run (no public drama required).
3. For one critical mart, write the freshness and volume checks you would run in hour 4. If they do not exist yet, file them as work items with the pipeline owner.
4. Agree with your manager on yellow versus red criteria in one short paragraph. Ambiguity is expensive during incidents.
5. Skim your quality checks and metric specs so row 6 of the board is not a scavenger hunt. Links: Data quality, Metrics.

Quick recap

• A wrong trusted number is a data incident for stewards, even when it is not a security breach.
• First 24 hours: contain, log, triage, run the check board, communicate on a clock, hand off with evidence.
• Severity is about decision harm and blast radius, not how loud Slack is.
• Check pipeline reality before you rewrite definitions.
• Clear the decision surface. Green jobs alone are not resolution.

Next in the series: Part 5 covers retention, deletion, and the seductive phrase “we might need it someday.” Architecture posts on governance and master data management stay linked for program context. Pipelines still matter: How data actually moves.

Sources

Research and further reading used for this article:

• NIST: Cybersecurity Framework (incident response and risk communication habits that transfer to operational data incidents)
• CISA: Incident Response Plan Basics (contain, eradicate, recover style sequencing adapted here to wrong-number operations)
• NIST SP 800-61 Rev. 2: Computer Security Incident Handling Guide (classic handling phases; useful structure even when the “asset” is a trusted metric)
• NIST Privacy Framework (when wrong data and privacy risk overlap, use privacy risk language early)
• Analytics Made Simple: How data actually moves (freshness, volume, schema, tests)
• Analytics Made Simple: Data quality (checks you should already have before the fire)
• Analytics Made Simple: Data governance (program layer this series practices against)

Merch on Etsy

Open store →

Mute Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, You're On Mute View on Etsy → Good Cheap Fast White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy →