The chart is clean. The SQL ran. The deck looks sharp. And somehow the room still makes a worse decision than if nobody had opened a laptop. That is not always a data quality problem. Sometimes it is a thinking trap wearing a dashboard costume.
This is a short companion to Analytics foundations, especially Reading a number like an adult and Data vs information vs insight. You will learn the traps that show up most often at work, how to spot them in five seconds, and what to say instead of “looks fine to me.”
What you’ll learn
- Survivorship bias in product and funnel data
- Cherry-picking windows and segments
- Averages of averages (and why store-level rates lie)
- Wrong grain, silent exclusions, and missing baselines
- A one-page trap checklist for reviews
Five traps that fake certainty
Example:
1. Survivorship bias
What it is: You study only the things that “made it” into the table. Failed pilots, churned customers, killed SKUs, and rejected applicants never show up, so the remaining set looks smarter, healthier, or more successful than reality.
Workplace version: “Our power users love feature X” when power users are defined as people still active after 90 days. The people who hated X already left. You are interviewing the survivors.
What to ask: “Who is missing from this dataset by construction?” “What would we see if we included canceled, deleted, or never-activated?”
2. Cherry-picking
What it is: Choosing the window, segment, or metric that tells the story you already wanted. Not always evil. Often unconscious. Always dangerous in a decision meeting.
Workplace version: “Revenue is up 18%!” (week of a one-time enterprise deal). Or “Conversion improved” (after dropping mobile traffic from the denominator without saying so).
What to ask: “Show me the same metric last year same week, last 12 weeks, and without the one-off.” If the story only lives in one cut, it is a cherry.
3. Average of averages
What it is: Averaging group-level rates as if each group were the same size. A store with 10 customers and a store with 10,000 customers are not equal votes.
| Store | Customers | Conversion |
|---|---|---|
| A | 100 | 10% |
| B | 100 | 10% |
| C | 10,000 | 2% |
| Naive average of rates | (10+10+2)/3 = 7.3% | |
| Customer-weighted rate | ~2.2% |
The naive average says “about 7%.” The business lives near 2%. If you present 7%, someone will staff and spend as if you had a miracle funnel.
What to ask: “Is this weighted by volume?” “Can I see overall rate and distribution, not only the mean of group rates?”
4. Wrong grain
What it is: Mixing what one row means. Counting order lines as orders. Counting sessions as users. Joining until the row count explodes and “revenue” doubles.
Workplace version: A join to a marketing attribution table multiplies each order by the number of touchpoints. Suddenly “orders” are up 3x and nobody changed the product.
-- Smell test: if a join multiplies rows, aggregates can lie.
-- Count distinct keys before and after the join.
SELECT
COUNT(*) AS row_count,
COUNT(DISTINCT order_id) AS orders,
SUM(amount) AS revenue_sum
FROM analytics.orders_joined_touches;If row_count is much larger than orders, you are one careless SUM away from fiction. Fix grain before you fix the slide title.
5. No baseline (“compared to what?”)
What it is: A lonely number with no prior period, plan, peer, or counterfactual. Covered in depth in reading numbers like an adult. It still earns a spot here because it is the most common trap in Slack.
Workplace version: “We hit 12,000 signups.” Without visits, last year, plan, or quality, you cannot know if you should celebrate or investigate.
What to ask: Always force one baseline: vs last year, vs plan, or vs a peer segment. Two is better. Zero is theater.
Honorable mentions (still common)
| Trap | Smell | Fix |
|---|---|---|
| Percent vs points | “Up 50%” from 2% to 3% | Say both points and relative change |
| Simpson’s shadow | Overall up, every segment down (or reverse) | Show mix shift + segment rates |
| Seasonality amnesia | Dec vs Nov “growth” | Same period last year |
| Silent definition change | Metric jumps on a Tuesday | Changelog for metric definitions |
| Overfitting the anecdote | One customer story drives strategy | Quantify how common the story is |
A review checklist you can paste into a PR or deck
TRAP CHECK (before this number leaves the room)
[ ] Who is missing from the data by design? (survivorship)
[ ] Would another time window reverse the story? (cherry-pick)
[ ] Are rates volume-weighted? (avg of avgs)
[ ] Does one row mean what we think? (grain)
[ ] Compared to what? (baseline)
[ ] Definition same as last month?
[ ] Exclusions listed in plain language?
[ ] Decision this number should change: ________How traps connect to the foundations series
- Problem first: traps thrive when the decision is fuzzy. Use a one-page brief.
- Data vs insight: a trap-free chart can still be only information. Insight changes a decision.
- Good enough: do not wait for perfect data to avoid traps. Many traps are reasoning errors on fine data.
- Reading numbers: rates, denominators, and baselines are trap detectors.
How to practice this week
- Take one chart from last week’s meeting. Run the trap check. Write which trap was closest.
- Rebuild one average-of-averages metric with a weighted rate. Compare.
- For one “success” cohort, estimate who never entered the table.
- Add a “compared to what” line to your next Slack metric update.
A mini scene: the “record conversion” that was not
Marketing posts a win: conversion jumped from 2.1% to 3.4% after a landing page rewrite. Leadership wants to scale the same design to every campaign. Adult trap check:
- Cherry-pick: the test week included a brand TV spot. Remove those days and the lift halves.
- Survivorship: the report only includes visitors who accepted cookies and completed a bot check. Mobile Safari users dropped out of the sample.
- Avg of avgs: country-level conversion rates were averaged equally; the US (huge volume, modest rate) was drowned by tiny high-rate geos.
- Grain: “conversion” mixed newsletter signups and paid checkouts in one funnel step.
- Baseline: no same-week last year, no holdout, no plan target. “Up” floated free of context.
None of those issues required a PhD. They required a habit: treat every shiny number as a suspect until the trap check passes. Scaling the page without that habit would have burned budget on a story, not a mechanism.
What to say in the room (scripts)
- “Before we decide, who is missing from this table by design?”
- “Can we see the same cut without the one-off week?”
- “Is this rate volume-weighted, or an average of group rates?”
- “What does one row mean after the join?”
- “Compared to last year, plan, or peer, which baseline should we use?”
These lines sound slightly pedantic the first time you use them. They sound like leadership the tenth time, when they have already saved a quarter.
Simpson-ish mix shifts without the textbook
You do not need the full Simpson paradox lecture to get hurt by mix. Overall conversion can rise while every channel falls if traffic shifts toward a higher-converting channel, or the reverse. The trap is presenting the overall as proof that the product got better when the mix did the work.
Habit: when leadership asks whether we are better, show overall and at least one segmentation that matters (channel, plan, region). If overall and segments disagree, lead with the disagreement, not the prettier chart. Mix shift is not a niche stats trick. It is how portfolios, funnels, and geographic rollups lie to busy people every week.
Definition drift: the quiet cousin of cherry-picking
Last month active user meant a login. This month it means any API call including background sync. The metric jumps. Someone claims a growth miracle. Nobody shipped a growth miracle. Somebody shipped a definition change without a changelog.
Treat metric definitions like code. If you cannot point to a written definition and a date it changed, you are flying without instruments. Pair this with the one-page brief grain field so active is never a vibe word in a decision document. Silent definition changes are how two honest analysts become permanent enemies over numbers that never meant the same thing.
Building a culture that catches traps early
Trap checks work better as norms than as personal pedantry. Require a compared-to-what line on every metric in a decision deck. In code review, ask for row counts before and after joins on any financial aggregate. Keep a living doc of metric definitions with owners and last change date. Reward people who kill bad charts early, not only people who ship more charts.
None of this slows a healthy team. It slows a theater team, which is the point. If your culture treats every question as a threat, fix the culture. If your culture treats every number as innocent until proven guilty, you will keep funding accidents. For deeper reading on rates, baselines, and denominators, keep the foundations series open. For cleaner asks before the traps appear, use the one-page brief. Traps are cheaper to prevent at the question stage than to unwind in a board meeting with twelve people and a frozen smile.
Quick recap
Clean data can still support dirty conclusions. Watch for survivors only, cherry-picked windows, unweighted averages of rates, wrong grain, and lonely numbers with no baseline. A short checklist beats a long argument after the deck is already in the meeting.
For templates that prevent fuzzy asks, use the one-page analytics brief companion. For the full foundations arc, start at the series landing.
Sources
- Classic discussion of survivorship bias in analysis (Abraham Wald / WWII aircraft armor as a teaching story; many modern summaries, e.g. Wikipedia overview): https://en.wikipedia.org/wiki/Survivorship_bias
- OpenIntro Statistics on averages, weighted means, and proportions: https://www.openintro.org/book/os/
- Analytics Made Simple foundations: problem framing, data vs insight, good enough data, reading numbers, analytics loop
