Peeking, guardrails, and stopping rules

Monday morning. The experiment dashboard is green. Someone posts a screenshot in Slack: “p < 0.05, ship it?” By lunch the number has faded. By Wednesday it is red. On Friday the team debates whether the week-two lift was real or whether three people refreshing a chart invented a win. Nobody set a stop rule. Everyone peeked.

Peeking is not “looking at data,” which is healthy. Peeking is treating every interim p-value as a final decision without a plan that allows interim looks. Guardrails are the metrics that can kill a “winning” primary. Stopping rules are the pre-agreed actions when the world is green, red, or muddy. Together they turn experimentation from a vibe into a decision system.

This is Part 2 of Experimentation culture. Part 1 covered design beyond the p-value: hypothesis, primary metric, sample, and units. Here we protect that design while the test is live. If you want the statistical intuition behind group comparisons and A/B tests first, the metrics series and the stats posts that feed this culture series pair well with what follows. For a broader path through analytics craft, start at Learn.

What you’ll learn

What peeking actually costs when people stop “when it looks good”
Fixed horizon vs planned sequential looks (safer patterns)
How to pick and enforce guardrails that can abort a win
Stopping rules: ship, stop/rollback, extend or cut
A worked experiment card with toy numbers and a practice drill

Peeking is optional looks without optional rigor

A classic fixed-horizon A/B test assumes you choose sample size (or duration), wait, then analyze once. The p-value and confidence interval are calibrated for that one look. If you open the dashboard every day and claim victory the first morning the p-value dips under 0.05, you have run a different experiment than the one you powered. Early green spikes are common under noise. Stopping on them inflates false positives.

That does not mean you must fly blind. Ops need health signals. Product needs a sense of trajectory. The difference is whether interim numbers can change the decision under a written plan, or whether they only feed curiosity and risk monitoring.

Three risky habits show up constantly:

Daily p-hunting: treating the primary metric’s p-value as a live scoreboard that can end the test any day.
Stop when green: ending early only when the variant looks good, never when it looks bad (one-sided peeking).
Ignore guardrails: shipping a conversion lift while refunds, latency, or support tickets are on fire.

Three safer habits mirror them:

Fixed horizon: analyze the primary at a pre-set date or sample size (ops monitors are separate and non-decision for the primary).
Sequential plan: if you will look early, use methods designed for multiple looks (group sequential, alpha spending, or platform sequential testing with documented settings).
Guardrail abort: pre-define metrics and thresholds that stop or roll back regardless of primary “wins.”

Filled peeking vs planned looks: risky daily p-hunting versus safer fixed horizon and guardrail abort

If your company uses an experimentation platform with sequential testing turned on, read the docs for that platform. “Always peeking is fine because the tool does sequential” is only true if you configure and interpret it as sequential. Using a fixed-horizon power calculator and then sequential-stopping in the tool is a mixed recipe.

Fixed horizon without theater

Fixed horizon is the simplest culture to enforce. You write: “We analyze primary on day 14 or after 40,000 users, whichever comes first, using the pre-registered metric and unit.” Until then, the primary chart may be visible with a banner that says “not decision-ready.” Ops still watch crash rates and payment failures.

Why people hate fixed horizon: it feels slow when a change “obviously” works. Sometimes it does. The cost of waiting two more days is usually smaller than the cost of shipping a fake lift and building three more features on a lie. Culture is choosing which cost you pay more often.

Practical fixed-horizon tips:

Pick a horizon that covers full business cycles (weekdays vs weekends, paydays, seasonal spikes).
Do not end early because a stakeholder is presenting on Thursday. Move the presentation or show “in progress, not decision-ready.”
Separate “health dashboards” (can trigger rollback) from “primary success dashboards” (trigger only at planned analysis).
Document the analysis unit (user, account, order) so nobody switches mid-test to get significance.

When sequential looks are the honest path

Some products cannot wait. High risk changes need early abort. High traffic teams want early stop for huge wins to free traffic for the next test. Sequential designs exist for that. They spend alpha (false positive budget) across looks, or they use Bayesian decision thresholds that you still must pre-commit to.

Minimum sequential plan contents:

How many looks (for example day 7, day 14, day 21).
What statistic and threshold apply at each look.
Whether early stop is allowed for success, futility, or both.
Who can call the stop (analyst + product owner, not whoever is loudest in Slack).
What happens to the primary if you stop for a guardrail instead.

If you do not have a statistician or a platform that implements this cleanly, prefer fixed horizon for primary decisions and keep early looks for guardrails only. That is not “less advanced.” It is matching ambition to tooling and skill.

Guardrails: the metrics that can veto a win

A primary metric answers the hypothesis. Guardrails answer “did we break something important while chasing that hypothesis?” Common guardrails: refund rate, chargebacks, latency p95, crash rate, unsubscribe rate, support tickets per user, margin, or a secondary funnel step that must not collapse.

Good guardrails are:

Few: three is usually better than twelve (noise multiplies with many thresholds).
Directional and actionable: “refund rate must not worsen by more than X absolute points” beats “monitor all metrics.”
Pre-registered: written before launch, not invented after the primary looked awkward.
Owned: named person who gets paged if the guardrail trips during the test.

Guardrail aborts are not failures of experimentation. They are the system working. Shipping a “winning” checkout button that triples failed payments is not a win; it is a finance incident with a chart attached. For metric definition discipline that makes guardrails computable, lean on patterns from the metrics series and quality checks from the data quality series.

Hard vs soft guardrails

Hard guardrails auto-stop traffic or force rollback (payment success, severe latency, safety).

Soft guardrails trigger a review, not an auto-kill (mild support volume uptick, secondary engagement dip). Soft is fine if the review is real. Soft that becomes “we noted it and shipped anyway every time” is theater.

Stopping rules that fit on one card

At analysis time you need three decision buckets, not a 40-slide debate:

Primary wins + guardrails OK → Ship (or graduated rollout with monitoring).
Guardrail red → Stop / rollback even if primary is green.
Inconclusive → Extend or cut under rules you wrote before, not under hope.

Stopping rules table: primary wins plus guardrails OK ships, guardrail red stops or rolls back, inconclusive extends …

“Extend” needs a cap. “We will add one more week once” is a plan. “We will keep going until it is significant” is peeking with a longer rope. “Cut” means you accept no ship, document learning, and free traffic. Inconclusive is a valid outcome. Treating it as personal failure creates pressure to mine segments until something green appears.

Worked example: checkout step reorder

Toy product: mid-size ecommerce. Hypothesis: putting shipping estimates earlier increases completed checkouts without hurting payment success.

Primary: checkout completion rate per started checkout (user-level assignment, 14-day fixed horizon, min 30k users per arm if traffic allows).

Hard guardrails: payment success rate must not fall more than 0.3 absolute points; p95 checkout latency must not rise more than 200ms.

Soft guardrail: support tickets tagged “checkout” per 1,000 starts must not rise more than 15% without a documented review.

Interim policy: no primary decision before day 14. Ops dashboard for latency and payment is live. If hard guardrail trips two consecutive days, auto-pause and investigate.

At day 14, toy results:

Metric	Control	Variant	Note
Checkout complete	62.1%	63.8%	Primary green, planned analysis
Payment success	97.4%	97.5%	Guardrail OK
p95 latency	1.8s	1.9s	Within +200ms
Support tickets / 1k	4.0	4.2	Soft, mild

Decision under the card: Ship with a one-week post-ship watch on soft support. Counterfactual: if payment success had been 96.9% against 97.4% control, decision is Stop / rollback despite a greener primary. That is the whole point of guardrails.

A second counterfactual people hate: primary is +0.2 points with wide uncertainty at day 14. The plan said “extend once to day 21 only if sample < power target; otherwise cut.” You extend under that clause, not because the CEO likes the design.

# Experiment decision card (YAML sketch)
experiment_id: checkout_shipping_early_v3
primary:
  metric: checkout_complete_rate
  unit: user
  analysis: fixed_horizon_day_14
guardrails:
  - name: payment_success
    type: hard
    rule: "variant - control >= -0.003 absolute"
  - name: checkout_p95_latency_ms
    type: hard
    rule: "variant - control

Segment peeking is still peeking

Related failure mode: primary flat overall, but mobile Safari in one country is green, so the team ships only there or claims “partial win.” Unplanned segment hunting multiplies false positives the same way daily looks do. Pre-register one or two segments you care about. Treat everything else as exploratory, labeled as such, and usually as fuel for the next experiment, not today’s ship.

If your culture rewards finding green cells in a 20-by-20 slice table, you will get green cells. You will not get reliable products. Part 3 of this series covers review meetings that force a decision without letting slices rewrite the primary after the fact.

Common mistakes

Powering for fixed N, stopping on first green day. Wrong procedure for the math you claimed.
Only peeking for success, never for futility. Biases toward shipping noise.
Guardrails invented after results. Becomes motivated reasoning with extra steps.
Twenty guardrails. Something will trip by chance; nobody respects the abort.
“Inconclusive means keep running forever.” Opportunity cost is real; cut and learn.
Changing the primary mid-test because the original metric “wasn’t moving.” That is a new experiment.
Hiding the interim chart from ops so hard failures go unnoticed. Safety visibility is not peeking.

How to practice this week

Pick one live or recent experiment. Write the three stopping buckets in five lines.
List current “guardrails.” Mark hard vs soft. Delete the ones nobody would actually abort for.
Check whether anyone can ship based on a mid-test screenshot. If yes, add a decision-ready banner and a named analysis date.
Simulate one guardrail-red primary-green case in a table. Practice saying “rollback” out loud with the product owner.
Optional: if you use sequential testing, open the platform docs and write which alpha-spending or sequential method is on. If you cannot find it, treat decisions as fixed-horizon until you can.

Quick recap

Peeking without a plan inflates wins. Fixed horizons and real sequential methods are both valid; mixing them casually is not. Guardrails protect the business from clever primaries. Stopping rules turn analysis into ship, stop, extend, or cut. Write them before traffic starts. Then stick to them when the Slack emoji party begins.

Next in Experimentation culture: experiment review meetings that decide, so these rules show up in a room with a clock and a decision log, not only in a YAML file nobody opened.

Sources

Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments (Cambridge University Press): practical peeking, guardrails, and decision culture: https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7ECBD9E84D
Johari, Pekelis, and Walsh on continuous monitoring and always-valid inference (Peeking at A/B Tests): https://arxiv.org/abs/1512.04922
FDA guidance on adaptive designs for clinical trials (group sequential concepts in plain regulatory language): https://www.fda.gov/regulatory-information/search-fda-guidance-documents/adaptive-design-clinical-trials-drugs-and-biologics-guidance-industry
Microsoft Experimentation Platform materials on metrics and guardrails (industry patterns): https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/
NIST Engineering Statistics Handbook, sequential methods overview: https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc42.htm