Experiment design beyond the p-value

A team can run a statistically tidy test and still make a bad product decision. Picture a test where the p-value came in under 0.05. (A p-value is a number that says how surprising the results would be if nothing had really changed.) The metric was a stand-in that nobody uses in planning. Only 8% of customers were even eligible to see the change. The rollout plan was to flip it to 100% on Friday, and the legal team first heard about the new wording on social media. The p-value did its narrow job, but the organization around it did not.

Experimentation culture is what happens around the calculator. It covers how you choose questions, protect users, write down designs, and decide when you are unsure. It also covers what you learn when the answer is “no lift.” Some statistics knowledge matters. Once your team understands the basics of an A/B test, process and incentives matter more. (An A/B test shows half your visitors version A and half version B, then compares what they do.)

This post opens a series on experimentation culture, written for analysts, product managers, and team leads who already know what an A/B test is and still watch good methods lose to calendar pressure. If you want the statistics backbone first, the Statistics for analysts series covers sampling, group comparisons, uncertainty, and A/B intuition. Here we go beyond the p-value on purpose.

What the p-value is for (and where it stops)

In a standard test, a p-value answers one narrow question. If there were truly no difference between the two versions, how surprising would data at least this extreme be? That assumes the math behind the test held up. That is all the p-value tells you. It does not give the chance that “no difference” is true. It says nothing about whether you made a good business decision, or how big or how important the effect is. The American Statistical Association (ASA) published a statement on p-values that is worth rereading each year if you live in experiment dashboards.

A healthy team treats the p-value as one input among several. The same goes for its cousins, such as Bayesian probabilities, which say how likely each version is to be better. Sequential stopping boundaries work the same way. They are pre-set lines that tell you when you may stop early. Put the number next to these:

• Estimated lift in business units (dollars, hours, retention points)
• The uncertainty range around the lift (the earlier statistics post on uncertainty explains how to read one)
• Implementation and reverse cost
• Guardrail health and user experience risk
• External validity (who was eligible vs who will get the ship)

Rule of thumb: If your only ship rule is “p less than 0.05,” you have handed your product judgment to an old habit of the statistics world.

Design starts with a decision, not a variant

Before wireframes, write the decision you are trying to inform:

• What will we do if the treatment wins by a lot?
• What will we do if it wins by a little?
• What will we do if it is flat?
• What will we do if it loses or guardrails trip?

If all four answers are “we will argue in a meeting,” you are not ready to spend traffic. Experiments use up a scarce resource. That resource is a pool of similar users who have not already been influenced by a half-finished change. Spend them on questions that can change your roadmap.

The experiment brief (minimum viable rigor)

Require a short brief before launch, because one page that people actually read beats a 40-slide deck nobody updates.

Link metric definitions to your trusted contracts (see the metrics series). If the main metric is not already a number your company tracks, you are testing two things at once: the product change and a brand-new way of measuring. Two unknowns at once is how muddy results pile up.

Decision rules that survive contact with leadership

Replace the question “is it significant?” with a policy everyone agreed to before the test. Here is one example.

• Ship: the range for the main lift sits mostly above +1.0 percentage point, the safety metrics stay within agreed limits, there are no serious group-size or data-quality flags, and a look at the major customer groups shows no severe harm.
• Iterate: the result points the right way but falls below the ship bar, or some groups gain while others lose and you have a clear next idea to try.
• Kill: the main metric goes down, a safety metric is breached, or the data quality fails.
• Extend: only if the written plan allowed extending the test, and never just because the chart is almost green.

Write numbers that match the business. A 0.2 point lift on a small, rarely visited page may not pay for the engineering time, while the same lift on the checkout page of a large retailer might fund a team for a year. An effect size without that context is just another empty ritual, the same way a p-value without context is.

Ethics and “should we?” before “can we?”

Being able to randomly assign people to a change does not mean you should. Think through these four points.

• Harm and fairness: Does the test hold back a safety improvement, an accessibility fix, or a clear price disclosure from the comparison group for longer than needed?
• Informed expectations: Tricks that push people into choices they did not intend can “win” a test and still lose customer trust. These are often called dark patterns. Your team needs a way to say no that does not hurt the careers of the people who use it.
• Sensitive attributes: Splitting results by race, health, or other protected details may be restricted or need a specialist to review it, and curiosity is not a legal reason to do it.
• Data minimization: Record what you need for the decision and the quality checks, and do not keep a permanent pile of experiment leftovers with no purpose.

Write the ethics check into the brief with a name and a date. A light process up front beats a dramatic complaint after launch. The habits from the data stewardship series apply here too, because an experiment is data collection that changes what real people see.

Quality checks: group sizes, A/A tests, and boring checklists

Experimentation programs that people trust are a little boring on purpose. They run A/A tests, where both sides see the identical page, to prove the testing tool works. They check for sample ratio mismatch (SRM), which means the two groups came out a different size than planned. They confirm the tool really logged who saw what before anyone celebrates a lift. They keep a kill switch handy. When the data pipeline is broken, they refuse to read the results at all.

Analysts should own a quality check that runs before anyone reads the results.

1. Group sizes compared with the plan
2. Whether the change showed up correctly for a few real accounts
3. Fresh data, and metric definitions that match the brief
4. Bot and fraud filters applied the same way to both groups
5. Any site-wide change at the same time that could drown out the test

If the check fails, the win for your culture is stopping, because telling a story around a broken test only spreads the damage.

Worked example: a pricing page design package

Say your software company wants to test a simpler pricing grid. Leadership likes bold launches, so you slow the room down with a brief.

Problem: Prospective buyers say they are confused between the Pro and Business plans, and the sales team says deals stall while people pick a plan.

Hypothesis: If we cut the page down to three clearer plan cards with one recommended plan, more visitors will start a free trial from the pricing page, because they will spend less time stuck comparing tiny feature footnotes.

Decision policy (written before launch):

• Ship if the trial-start rate rises by at least 1.5 percentage points with the range mostly above +0.8, paid conversion within 14 days is not down by more than 0.3 points, and support tickets tagged “pricing confusion” do not rise.
• Iterate if trial starts rise but paid conversion softens, since the new message may be attracting people who are less serious about buying.
• Kill if trial starts fall or the paid conversion safety limit is crossed.

Population: anonymous and logged-out visitors to the /pricing page who use the English-language site, leaving out existing paid accounts. That is about 22% of site traffic, and saying so keeps anyone from assuming the result also applies to upgrade prompts inside the app.

Unit: the browser identity that the testing tool uses, counted per visitor for trial starts, with paid conversion counted per account among the people who created one.

Runtime: sized so the test has an 80% chance of catching a 1.5 point lift (the smallest lift worth detecting, or MDE) from a 6% starting trial-start rate. At current traffic that takes about three weeks. Nobody stops early for a win unless the tool’s written stopping lines say so.

Ethics and brand: both versions show accurate prices and yearly totals with no hidden fees, someone checks that keyboard users can tab through the new cards in a sensible order, and legal signs off on the claim wording.

Here is a made-up final readout to show how the policy gets applied.

Here is where culture shows up. Even with a clean result, you roll out in stages of 25%, 50%, then 100%. You watch page speed and how visitors arriving from search behave. If the team wants to learn how the change holds up over months, you also keep a small share of visitors on the old page for four weeks. The p-value goes in the appendix of the readout, and the decision policy goes on page one.

A lightweight brief in Markdown you can paste into tickets

## Experiment brief: pricing grid v2

Problem: ...
Hypothesis: If we ..., then ... because ...
Decision:
  - Ship if ...
  - Iterate if ...
  - Kill if ...
Population / exclusions: ...
Unit of randomization: ...
Primary metric (definition link): ...
Guardrails: ...
MDE / power / stop rule: ...
Risks & owners: ...
Quality gate owner (SRM, exposure QA): ...

Learning from flat and negative results

Teams that only celebrate green tests teach people to cherry-pick numbers until something looks significant, which is called p-hacking. They also learn to shop around for a metric that moves and to bury failures. A flat result can mean the idea is weak. It can also mean the metric is not sensitive enough, the test ran too briefly, or the change was built poorly. A negative result can save years of roadmap. Keep briefs and outcomes in a place people can search, and treat “we killed it early when a safety metric broke” as good work.

Analysts can help by writing a review after each test that keeps four things apart.

• How good the product idea was
• How well it was built and measured
• Whether the test was big enough to notice a change
• How well the team made the decision

Each one is a different way to fail, so each needs a different fix.

Rollouts are part of the design

When the experiment ends, the product decision begins. A staged rollout catches problems that a 50/50 split did not show. Examples are stale cached pages, outside scripts that misbehave, and payment methods that only fail in some regions. A long-term holdout, meaning a small group that keeps the old version, shows whether the excitement of something new wore off. If several teams test at once, they also need an agreed way to share traffic. The people who run the testing tool and the analysts should publish a few simple rules. Say how many tests may run on the same page at once, how to set aside a holdout group, and who can override the rules.

Common mistakes

• Shipping on the p-value alone and ignoring the size of the effect and its cost.
• Launching without a written decision policy.
• Testing pushy tricks (dark patterns) because they “win.”
• Applying a result from a tiny eligible slice to the whole customer base.
• Skipping the quality checks when leadership wants a story for the board.
• Punishing teams for flat tests, which nearly guarantees they will shop for a friendlier metric next time.
• Switching everyone over at once after a short test on a page that matters a lot.
• Letting the tool’s default green badge stand in for the brief.

How to practice

1. Take one past test and fill in the brief template for it. Note what was missing when it launched.
2. Hold a 20-minute meeting that only agrees on the ship, iterate, and kill lines, before anyone designs a variant.
3. Add a quality checklist to your team’s definition of “ready to launch.”
4. Publish one write-up of a flat or negative result that treats the outcome as learning and not as shame.
5. Work out who can veto a test that is unethical or damages the brand. If the answer is “nobody,” fix that before the next growth brainstorm.

Series notes: This is Part 1 of the experimentation culture series. Later posts go deeper on how organizations run experiments, including review meetings, choosing a mix of bets, and explaining uncertainty to executives. For the statistics foundation, see the Statistics for analysts series. For more skills, visit Learn, and for metric definitions see the metrics series.

Quick recap

• A p-value is a narrow tool. Decisions also need effect size, cost, safety metrics, and scope.
• Start from the decision you will make, then design the test.
• A one-page brief fixes who is in the test, the metrics, the smallest lift worth detecting, the risks, and the owners.
• Ethics and brand vetoes belong in the design, not after it.
• Quality checks on group sizes, exposure, and definitions protect you from false stories.
• Staged rollouts, holdouts, and an archive of what you learned turn one readout into a habit.

Sources

• American Statistical Association. Statement on statistical significance and p-values. https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• Kohavi, Tang, and Xu. Trustworthy Online Controlled Experiments (culture, metrics, decision making). https://experimentguide.com/
• Microsoft Experimentation Platform research on trustworthy online experimentation practices. https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/
• UK Government Service Manual. “Measuring success” and related guidance on ethical, user-centered measurement (public-sector clarity). https://www.gov.uk/service-manual/measuring-success
• National Institute of Standards and Technology (NIST) Privacy Framework (for thinking about data practices around experimentation logging). https://www.nist.gov/privacy-framework

Merch on Etsy

Open store →

Mean Squared Error Shirt View on Etsy → Apache Iceberg T-Shirt View on Etsy → You're On Mute Coffee Mug View on Etsy → Git Version Control Shirt View on Etsy →