Comparing groups without fooling yourself

Two cities. Two conversion rates. City A is at 4.8%. City B is at 3.1%. Someone draws a roadmap arrow toward “copy whatever City A is doing.” Nobody asks whether City A is mostly desktop renewals while City B is mostly new mobile signups after a local ad burst. The chart is accurate. The comparison is not fair.

Comparing groups is everyday analyst work: segments, regions, plans, cohorts, channels, versions. It is also where silent mix shifts, Simpson’s paradox vibes, and cherry-picked baselines invent strategy from noise. You do not need a PhD to avoid the common traps. You need a checklist and the discipline to show the denominator.

This is Part 2 of Statistics for analysts. Part 1 covered sampling and bias: who is in the data. This part assumes you have a sample and asks a harder question: when two slices disagree, what are you actually allowed to conclude?

What you’ll learn

How to state a comparison question so it has one clear unit and one clear outcome
Why mix shifts and composition changes fake wins
Baselines that do not lie: same window, same eligibility, same definition
A worked example that unmasks a “segment win” with stratified rates
When to stop at descriptive comparison and when you need an experiment

Write the comparison sentence before you query

Fill this template and refuse to ship the chart until it is complete:

Among [population], how does [outcome] differ between [group A] and [group B], measured over [time window], with unit [user/account/session], after applying [shared filters]?

If you cannot fill a slot, the query will invent an answer for you. Shared filters matter most. Comparing “all mobile” to “desktop enterprise renewals last Black Friday” is not a group comparison. It is two different studies stacked on one slide.

Link the outcome definition to your metric contract habits from the metrics series. Group labels are dimensions. Outcomes are metrics. Mixing informal labels (“engaged users”) with precise numerators is how meetings multiply.

The three things that move a group rate

When Group A’s rate is higher than Group B’s, at least one of these is true:

Composition: A has a different mix of subtypes (more enterprise, more returning, more US).
Behavior or treatment: something about A actually performs differently inside comparable subtypes.
Measurement: tracking, eligibility, or timing differs by group.

Strategy wants (2). Dashboards often show (1) dressed as (2). Instrumentation quietly contributes (3). Your job is to separate them far enough that a product manager does not ship a rewrite based on a mix shift.

Diagram showing overall rate split into composition mix and within-group rates for fair comparison

Apples to apples: the fairness checklist

Same time window and seasonality context

Do not compare this week’s new feature cohort to last year’s holiday cohort without saying so. Align windows, or use multiple windows and show stability. If one group only exists after a launch date, the other group’s pre-launch history is not a twin.

Same eligibility

Both groups should pass the same “could have been measured” rules. If Group B includes users who never saw the paywall because of a geo block, conversion-to-paid is not the same experiment as Group A where everyone saw it.

Same unit and grain

Session conversion vs user conversion will disagree. Account-level retention vs seat-level activity will disagree. Fix grain first (see also key-term thinking on grain elsewhere on AMS). One row meaning one honest thing is non-negotiable for comparisons.

Same metric definition

If finance’s “revenue” includes tax in one export and excludes it in another, your region comparison is a taxonomy fight. Pin the definition ID or the dbt model name in the footnote.

Enough volume in each cell

A 50% conversion rate on 8 users is a story about 4 people, not a strategy. Show counts next to rates. Hide or gray out cells below a minimum n you agree with stakeholders. Part 3 will talk about uncertainty more formally. For now, refuse to rank tiny slices.

Mix shifts: the silent product manager

Overall conversion can rise while every segment is flat, if high-converting segments grow as a share of traffic. Overall conversion can fall while every segment improves, if low-converting segments grow faster. That is not a paradox for people who check composition. It is Tuesday.

Always carry two views:

Overall rate for the business scoreboard.
Segment rates plus segment mix for diagnosis.

If leadership only sees overall, they will reward traffic composition changes as product genius. Sometimes acquisition strategy really did improve. Say that clearly. Do not call it a checkout redesign win if checkout segment rates did not move.

Standardization in plain English

When mix differs, you can reweight group rates to a common mix (a simple form of standardization). Example: hold both regions to the same plan-tier mix, then recompute a weighted conversion rate. You are asking: “If the mix were the same, would the gap remain?” If the gap shrinks a lot, composition was doing the heavy lifting. If the gap remains, look for within-segment differences or measurement issues.

You do not need perfect epidemiology methods on day one. A spreadsheet with segment rates and a shared weight column already prevents most self-deception.

Worked example: the “Pro plan is better” slide

Claim from growth: Pro plan users convert to annual billing at 22%, Basic at 11%, so push everyone to Pro. You pull a fairer cut. Toy numbers for teaching:

Plan	Segment	Users	Annual conversions	Rate
Basic	New self-serve	8,000	640	8%
Basic	Returning	2,000	400	20%
Basic	All Basic	10,000	1,040	10.4%
Pro	New self-serve	1,500	135	9%
Pro	Returning	3,500	875	25%
Pro	All Pro	5,000	1,010	20.2%

Overall, Pro looks twice as good. Inside new self-serve, Pro is only one point higher (9% vs 8%). Inside returning, Pro is higher (25% vs 20%), but Pro’s mix is mostly returning users. If you force new self-serve users onto Pro without changing who they are, you should not expect the overall 20% rate to transfer.

A standardized story: put both plans on Basic’s mix (80% new, 20% returning).

Basic standardized: 0.8×8% + 0.2×20% = 10.4% (same as overall Basic by construction here).
Pro standardized to Basic’s mix: 0.8×9% + 0.2×25% = 12.2%.

The gap shrinks from about 10 points overall to about 2 points after mix alignment. That is still a gap worth studying. It is not “Pro doubles annual conversion for everyone.” The product decision changes: maybe improve returning-user motions on Basic, or test Pro’s value props on new users with a real experiment, rather than a global push based on a blended rate.

Result table showing overall Pro vs Basic gap shrinking after standardizing to the same user mix

SQL sketch: rates with counts, not rates alone

SELECT
  plan_tier,
  user_segment,
  COUNT(*) AS users,
  SUM(converted_annual) AS conversions,
  ROUND(100.0 * SUM(converted_annual) / COUNT(*), 1) AS conv_pct
FROM annual_billing_eligible
WHERE eligible_date BETWEEN DATE '2026-01-01' AND DATE '2026-03-31'
  AND country IN ('US', 'CA', 'GB')  -- shared frame
GROUP BY 1, 2
ORDER BY 1, 2;

Always select counts. Future you will thank present you when someone asks “is that 25% on 12 people?”

Baselines that do not gaslight the room

Comparisons need a baseline story:

Peer baseline: Group A vs Group B in the same window.
Time baseline: this period vs prior period for the same group.
Target baseline: vs a goal you set in advance (not after seeing the data).

Changing the baseline after you see the chart is a way to always win. Pre-register the comparison in the ticket or the analysis plan: “We will compare week 12 to week 11 for the same eligibility, and we will also show US vs UK with shared filters.” If you later add cuts, label them exploratory.

When descriptive comparison is enough (and when it is not)

Descriptive group comparisons are enough when the decision is “where should we look,” not “what caused this.” They guide prioritization, QA, and hypothesis lists. They do not, by themselves, prove that changing Group B to look like Group A will transfer Group A’s outcome.

Causal claims need design: experiments (Parts 3-4 and the experimentation culture series), strong quasi-experiments, or domain knowledge so solid that the room accepts residual risk. If someone says “the data proves the blue button causes conversion,” and the blue button only shipped to logged-in desktop users in one country, you have a comparison, not a proof.

Common mistakes

Ranking segments on rate alone without n and mix.
Declaring a winner across unequal eligibility (one group never saw the offer).
Ignoring measurement differences by platform or region.
Using open-ended “engaged” groups defined after looking at outcomes (circular segments).
Multiple quiet cuts until a flattering slice appears, then presenting it as the primary analysis.
Comparing pre/post without checking concurrent changes (pricing, ads, outages).
Assuming within-group equality because overall rates match (hidden opposing segment moves).

How to practice

Take one “Segment X is better” claim from last month. Rewrite it with the comparison sentence template.
Add a mix table: share of traffic by subtype for each group.
Recompute one standardized rate with a shared weight vector in a spreadsheet.
Set a minimum n rule with a partner (for example, no rate callouts under 200 units without a caveat).
Label one analysis “descriptive prioritization” and one “causal claim needs experiment” so the language sticks.

Next: uncertainty and confidence in plain English, so “12% vs 10%” comes with a sense of stability. Browse the Learn hub for related paths. For pipeline and definition hygiene that keeps group labels trustworthy, see the data quality series.

Quick recap

Write the comparison sentence (population, outcome, groups, window, unit, shared filters) before SQL.
Overall gaps can be composition, true within-group differences, or measurement artifacts.
Show rates with counts and mix, not rates alone.
Standardize to a common mix when groups differ in composition.
Precommit baselines; do not shop for a flattering reference after seeing results.
Descriptive comparisons prioritize; causal transfer usually needs better design.

Sources

OpenIntro Statistics. Comparing groups and observational studies chapters. https://www.openintro.org/book/os/
Harvard Business Review / classic analytics writing on Simpson’s paradox in business settings; a clear technical walkthrough is available from the Stanford Encyclopedia-style explanations and many stats notes. Accessible overview: https://plato.stanford.edu/entries/paradox-simpson/
CDC principles of epidemiology: adjusted rates and confounding concepts transferable to business mix. https://www.cdc.gov/csels/dsepd/ss1978/lesson3/section5.html
Kohavi, Tang, and Xu. Trustworthy Online Controlled Experiments (why observational segment gaps are not automatic ship decisions). Book site and related materials: https://experimentguide.com/
UK Office for National Statistics methodology notes on comparing rates and compositions (public-sector clarity worth stealing). Example hub: https://www.ons.gov.uk/methodology