A/B tests intuition for analysts

The experiment tool shows a green checkmark. Lift is +3.2%. p-value is 0.03. Someone types “ship” in the channel. Then a quieter person asks whether we peeked every day for three weeks, whether the metric matches the decision, and whether mobile crashed while desktop carried the win. The green checkmark does not answer those questions. Your intuition for experimental design does.

A/B tests (online controlled experiments) are the cleanest way many product teams get causal evidence: randomly assign comparable units to variants, measure outcomes, and estimate the effect of the change you made. They are powerful. They are also easy to misuse when the culture wants certainty faster than the sample can provide it.

This is Part 4 of Statistics for analysts, and it closes the short stats arc: sampling (Part 1), fair comparisons (Part 2), uncertainty (Part 3), and now intentional experiments. The companion series on experimentation culture starts next with design beyond the p-value. Here we build the core intuition you need before the tooling debate begins.

What you’ll learn

What random assignment buys you (and what it does not)
Primary metrics, guardrails, and the unit of randomization
Power, MDE, and why “two weeks” is not a sample size strategy
Peeking, multiple testing, and novelty effects in plain language
A worked readout for a checkout experiment with a decision table
How analysts partner with PMs without becoming the Department of No

The experiment in one paragraph

You pick a population that is eligible for a change. You randomly assign units (users, accounts, pages, search queries) to control or treatment. You run long enough to collect a pre-planned sample (or use a sequential design that is built for peeking). You compare outcomes on a primary metric, check guardrail metrics for harm, and decide ship, iterate, or abandon based on effect size and uncertainty, not vibes alone.

Random assignment is the star. On average, it balances both measured and unmeasured confounders across variants. That is why a well-run A/B test can support causal language that a dashboard segment cut cannot. It does not fix a primary metric that does not match the business decision. It does not fix a sample of only power users. It does not fix peeking without a plan.

Rule of thumb: Randomization balances who is in each bucket. It does not balance bad metric choices, broken logging, or a stopping rule written after you liked the curve.

Filled A/B flow: eligibility, assignment, exposure, metrics, decision

Unit of randomization: the choice that haunts you later

If you randomize by page view but users see both variants across sessions, interference and inconsistent experience wreck the comparison. If you randomize by user but the decision is account-level billing, seats in the same company may get different treatments and support will hear about it. Common units:

User / cookie / device: common in consumer product UI tests. Watch multi-device users.
Account / organization: common in B2B. Fewer units, slower tests, cleaner commercial metrics.
Session: sometimes used; risky if learning or cart state spans sessions.
Geographic or time-based switches: more quasi-experiment than classic A/B; different assumptions.

Analysis should respect the unit. Treating events as independent when you randomized users understates uncertainty (Part 3). Align randomization unit, analysis unit, and decision unit as much as you can.

Metrics: primary, secondary, guardrails

Pick one primary metric that would justify the change if it moved enough. Examples: checkout completion rate, activated accounts per signup, weekly retained users. Write the definition the way you would in the metrics series: numerator, denominator, window, exclusions.

Secondary metrics explain how the primary moved (steps in a funnel, time to complete). They are for learning, not for an unlimited fishing license to declare victory.

Guardrails catch harm: error rates, latency, refund rate, unsubscribe rate, support tickets, revenue per user if you optimized a proxy. A primary win with a guardrail fire is not a clean ship. It is a tradeoff meeting.

Role	Job	Example
Primary	Decision metric	Purchase conversion among eligible users
Secondary	Mechanism / diagnosis	Add-to-cart rate, payment step completion
Guardrail	Do-no-harm	Checkout errors, refund rate, p95 latency

Power, MDE, and calendar-driven science

Minimum detectable effect (MDE) is roughly: the smallest lift you want to be able to detect with a planned power (often 80%) at a chosen false positive rate (often 5% two-sided, though teams vary). If your business only cares about lifts of 5% relative or more, designing for a 0.5% MDE wastes time. If you need to detect 1% relative on a noisy metric, you need more units or a longer runtime than a sprint calendar wants.

Saying “we will run it for two weeks” without checking traffic and baseline rate is hope-based methodology. Traffic is not time. A low-traffic B2B surface may need months. A top-of-funnel consumer change may need days. Do the size math or use your platform’s calculator before launch, and write the planned end condition down.

Also plan for variance reduction when ethical and available: CUPED-style methods, better stratification, or sharper metrics. These are advanced tools with assumptions. The beginner win is simply not stopping early because Wednesday looked good.

Peeking, multiple comparisons, and other ways to invent wins

Peeking

If you check p-values every hour and stop when you first see p < 0.05, you inflate false positives. Sequential testing designs exist for continuous monitoring. “We will look daily and stop whenever it is green” is not one of them unless your platform’s methodology says it is. Prefer pre-planned runtime or proper sequential boundaries.

Many metrics and many slices

Twenty secondary metrics and fifteen country cuts will produce stars by chance. Pre-specify the primary. Treat slices as exploratory unless you planned for them (and budgeted alpha or used hierarchical methods). If an exploratory slice looks huge, consider a follow-up experiment aimed at that hypothesis.

SRM and interaction bugs

Sample ratio mismatch (SRM): you expected 50/50 assignment and got 47/53 with a huge n. That is a quality alarm, not a curiosity. Trigger bugs, bot filters, and redirect errors cause weird results. Stop and debug before storytelling.

Novelty and primacy

Users may click a shiny change once, or resist a layout change for a week then adapt. Runtime should cover more than a single novelty day when the behavior is habitual. For some changes, long-run holdouts are the adult supervision your launch needs.

Worked example: checkout microcopy test

Hypothesis: clearer shipping cost copy on the payment step raises purchase conversion without raising refunds. Population: logged-in users in US/CA who reach payment. Unit: user. Primary: purchase within 24 hours of first payment-step view in the experiment. Guardrails: payment error rate, refund rate within 7 days. Planned runtime: to 80% power for a 2.0 absolute point lift from a 10% baseline (MDE set with stakeholders). Toy readout at the planned end:

Metric	Control	Treatment	Difference	Notes
Users	18,400	18,220	SRM check pass	Assignment near 50/50
Purchase rate	10.1%	11.8%	+1.7 pts	Primary; interval excludes 0 in platform readout
Payment errors	2.4%	2.5%	+0.1 pts	Guardrail OK
7-day refund rate	1.1%	1.0%	-0.1 pts	Guardrail OK
Mobile purchase rate	8.0%	8.2%	+0.2 pts	Exploratory slice; weak alone
Desktop purchase rate	12.0%	14.9%	+2.9 pts	Exploratory; drives most of lift

Decision conversation: overall primary looks positive with healthy guardrails. Most lift appears on desktop in an exploratory cut. Options: ship globally if the overall effect clears the business bar and mobile is not harmed; ship desktop-only if engineering can target; or run a short follow-up focused on mobile copy. What you should not do: declare “mobile loves it too” from a +0.2 point noisy slice, or restart the same test and stop at the first green hour.

Result decision card for checkout A/B test with primary lift and guardrails passing

A minimal analysis checklist in SQL spirit

SELECT
  variant,
  COUNT(DISTINCT user_id) AS users,
  COUNT(DISTINCT CASE WHEN purchased = 1 THEN user_id END) AS buyers,
  ROUND(
    100.0 * COUNT(DISTINCT CASE WHEN purchased = 1 THEN user_id END)
      / COUNT(DISTINCT user_id),
    2
  ) AS purchase_rate_pct
FROM exp_checkout_copy_exposure e
LEFT JOIN exp_checkout_copy_outcomes o
  USING (user_id, variant)
WHERE e.first_exposed_at >= TIMESTAMP '2026-09-01'
  AND e.first_exposed_at <  TIMESTAMP '2026-09-22'
GROUP BY variant
ORDER BY variant;

Pair this with SRM checks, guardrail queries, and the platform’s stats engine. SQL alone is not a full sequential testing framework. It is how you verify the inputs.

When not to A/B test

Not every change deserves a test. Legal and safety fixes may need to ship. Brand-new surfaces may lack traffic. Some changes are so large that “control” is not a meaningful product state for long. Sometimes a qualitative study or a staged rollout with monitoring is better. The cost of an experiment includes engineering time, delayed value, and cognitive load. Match method to risk, as Part 1 said for sampling stakes.

Common mistakes

Stopping at the first green p-value without a plan.
Primary metric chosen after seeing results.
Ignoring SRM and exposure bugs.
Randomizing at one unit and analyzing at another without care.
Declaring global truth from a narrow eligibility pool.
Optimizing a vanity metric that conflicts with revenue or retention guardrails.
Running underpowered tests and calling “no significant difference” proof of equality.

How to practice

Write a one-page design for a fake test: population, unit, primary, guardrails, MDE, runtime rule.
Audit one past test: was the primary pre-specified? Any peeking? Any SRM?
Compute rough sample size with your platform calculator for a metric you care about.
Add a decision table template (ship / iterate / kill) to your team wiki.
Read the first chapter materials from a trustworthy experimentation guide (sources below) and map their vocabulary to your tool’s UI labels.

This ends the Statistics for analysts mini-series. Continue with Experimentation culture starting at experiment design beyond the p-value, where process, ethics, and decision quality take center stage. For more learning paths, visit Learn. Related metric hygiene: metrics series.

Quick recap

A/B tests use random assignment to support causal claims inside the eligible population.
Align randomization unit, analysis unit, and decision unit.
Pre-specify primary metrics, guardrails, and stopping rules.
Size tests to an MDE that matches business value, not only a calendar.
Peeking and slice fishing create fake wins; SRM flags broken plumbing.
Decisions need effect size and risk, not only a p-value color.

Sources

Kohavi, Tang, and Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. https://experimentguide.com/
Microsoft exp-platform papers and guides on trustworthy experimentation (SRM, metrics, culture). Starting point: https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/
Google “Overlapping experiment infrastructure” and related essays on large-scale experiments (historical context). Search via Google Research publications hub: https://research.google/pubs/
American Statistical Association statement on p-values (limits of significance language). https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
OpenIntro Statistics. Hypothesis testing chapters for baseline vocabulary. https://www.openintro.org/book/os/

Keep going

Same lessons in your feed

Short diagrams and hooks on Instagram, X, and Facebook.

@analyticsmadesimple Follow @AnalyticsSimply Follow Analytics Made Simple Follow