Experiment review meetings that decide

The calendar invite says “Experiment readout, 30 minutes.” Twenty-eight minutes later you have seen eight charts, three segment rabbit holes, and a promise to “circle back offline.” Nobody said ship, iterate, or kill. The variant stays at 50% traffic because indecision is quieter than a bad call. The next sprint still plans features on top of a maybe.

Experiment review meetings exist to produce a decision and a record, not a tour of every cut. Part 2 of this series gave you peeking rules, guardrails, and stopping buckets. Part 3 puts those into a room with a clock: result, segments, guardrails, decision log. If the meeting cannot end with one of ship / iterate / kill (plus who owns the follow-up), it was a status update wearing a lab coat.

This is Part 3 of Experimentation culture. It closes the loop from design (Part 1) and live-test discipline (Part 2) into organizational habit. Pair it with clear metric definitions from the metrics series and quality habits from the data quality series. For the broader learning map, see Learn.

What you’ll learn

A four-block flow: result → segments → guardrails → decision log
A 25-minute agenda that forces a decision
Roles in the room (who presents, who decides, who blocks)
How to handle inconclusive and “interesting segments” without rewriting the primary
A worked readout for a toy experiment and a reusable decision log template

What “decide” means

A decision is one of a small set of actions, with an owner and a date:

Ship: roll variant to 100% (or staged rollout with exit criteria), archive the experiment, monitor guardrails for a short post-period.
Iterate: keep learning with a new experiment ID, hypothesis, and plan. Not “leave 50/50 forever.”
Kill: remove the variant, document why, free traffic and engineering attention.
Extend (bounded): only if Part 2 stopping rules already allowed one extension; book the next review now.

“Interesting” is not a decision. “Needs more analysis” is only a decision if it names the question, the owner, and the 48-hour deliverable that unlocks ship/iterate/kill. Open-ended analysis is how experiments become permanent half-features.

The four blocks of a review that works

Run the meeting in a fixed order. Changing the order is how people hide bad news behind a fun segment story.

Filled experiment review: result, segments, guardrails, decision log

1. Result (primary first)

State the pre-registered hypothesis, primary metric, unit, sample, and planned analysis method in under two minutes. Then show the primary result with uncertainty (interval, probability of superiority, or whatever your stack uses), not only a point estimate. Say whether the planned stopping rule says win, loss, or inconclusive for the primary alone. No segments yet.

2. Segments (exploratory, labeled)

Show pre-registered segments only for decision weight. Exploratory cuts get a banner: “hypothesis generating.” If overall is flat and one cell is green, you do not rewrite the primary in the meeting. You may open an iterate experiment aimed at that segment with a fresh design.

3. Guardrails

Walk hard then soft guardrails. A green primary with a red hard guardrail is not a ship. If someone wants to “accept the risk,” that is an explicit product/risk decision recorded in the log, not a silent shrug.

4. Decision log

Write the decision while people are still in the room. Paste it into the experiment ticket, wiki, or platform notes the same hour. Future you will not remember the verbal agreement from a noisy Zoom.

A 25-minute agenda that respects calendars

Long meetings invite digressions. Short meetings force pre-work. Try this default:

Review agenda table: 5 minutes hypothesis and metric, 10 minutes result and uncertainty, 5 minutes guardrails, 5 minu…

5 min: Hypothesis + metric (and unit, horizon, peeking policy).
10 min: Result + uncertainty (primary; pre-registered segments only if time).
5 min: Guardrails.
5 min: Ship / iterate / kill (or bounded extend) + owners.

If the readout needs more than 25 minutes of math explanation, the pre-read failed. Send a one-pager 24 hours ahead: hypothesis, primary chart, guardrail table, recommended decision. The meeting is for questions and commitment, not for live SQL.

Roles in the room

Role	Job in the meeting	Not their job
Analyst presenter	Facts, uncertainty, plan compliance	Owning the business tradeoff alone
Product decision owner	Ship / iterate / kill call	Relitigating metric definition mid-call
Eng owner	Feasibility of rollout and rollback	Veto by endless “what if we also measure…”
Risk / ops (as needed)	Guardrail interpretation	Blocking every soft yellow forever
Scribe	Decision log live	Silent observer who “will write it later”

If the product decision owner is not in the room, you do not have a decision meeting. You have a rehearsal. Reschedule.

Pre-read packet (one page)

Required fields before the invite goes out:

Experiment ID, link to design doc, dates, traffic share.
Hypothesis in one sentence.
Primary metric definition (numerator, denominator, unit, window).
Stopping rule and peeking policy (from Part 2).
Primary result with uncertainty.
Guardrail table (pass/fail).
Recommended decision and why (two sentences).
Open risks (data quality, SRM, bugs) in plain language.

Sample ratio mismatch (SRM) and tracking bugs deserve a red banner at the top if present. You cannot out-debate a broken assignment pipeline with a clever narrative. Fix the data path, restart or re-run under a clean design, then review again.

Worked example: onboarding checklist experiment

Toy SaaS. Hypothesis: a three-item checklist on first login raises day-7 activation without hurting time-to-first-value for power users.

Primary: day-7 activation rate (user unit), fixed 14-day horizon, sequential not used.

Hard guardrail: support tickets per new user must not rise more than 10%.

Soft guardrail: median time to first key action must not rise more than 20%.

Pre-read summary the analyst sent:

EXPERIMENT: onboarding_checklist_v2
HYPOTHESIS: Checklist raises D7 activation without +support load
PRIMARY: d7_activation_rate (users), fixed day-14 analysis
RESULT: control 28.4% | variant 31.1% | 95% CI on lift +1.1 to +4.3 pp
GUARDRAILS:
  support_per_new_user: +4% (PASS hard)
  median_ttf_key_action: +8% (PASS soft)
SRM: none
RECOMMENDATION: Ship to 100%, monitor support 7 days
OPEN QUESTIONS: exploratory mobile lift larger; not decision-grade

Meeting flow (actual clock):

Minutes 0 to 5: product owner restates hypothesis; analyst confirms no mid-test primary change.
Minutes 5 to 15: primary result; someone asks for country cuts; facilitator parks to “exploratory parking lot.”
Minutes 15 to 20: guardrails green; eng confirms rollback flag still works.
Minutes 20 to 25: decision = Ship. Owner = product. Eng schedules full rollout Tuesday. Analyst files decision log. Parking lot item becomes a future experiment idea, not a ship condition.

Decision log entry:

date: 2026-10-15
experiment_id: onboarding_checklist_v2
decision: ship
rationale: >
  Primary lift positive at planned analysis; hard and soft guardrails pass;
  no SRM; rollback path verified.
owners:
  rollout: eng_onboarding
  post_ship_watch: analyst_growth
  product_signoff: pm_activation
follow_ups:
  - task: full_rollout
    due: 2026-10-16
  - task: support_watch_7d
    due: 2026-10-22
non_decisions:
  - mobile_segment_lift is exploratory only; may seed checklist_v3

Handling the hard cases

Inconclusive primary, clean guardrails

Default to kill or iterate, not infinite 50/50. Iterate only with a sharper hypothesis (bigger effect expected, better targeting, better UX). “Run longer until green” needs the Part 2 extend clause or it is peeking.

Primary win, soft guardrail yellow

Options: ship with a tighter post-ship watch, iterate a fix for the soft issue, or kill if the soft metric is nearly hard in business terms. Record the tradeoff. Do not pretend the yellow was green.

Primary loss, one glamorous segment win

Default kill the global variant. Optionally iterate with a segment-targeted design and power for that segment. Shipping a global loss because a slice looked good is how you bake noise into the product.

Data quality red flag mid-review

Stop the decision. The decision becomes “invalidate or pause; fix instrumentation; re-review.” Shipping on dirty assignment is worse than waiting a week. This is culture: truth over theater.

Anti-patterns in experiment meetings

Slide archaeology: 30 charts, zero decision sentence.
HiPPO override without log: leadership ships against the plan and nobody writes why.
Metric shopping live: “What if we used revenue instead?” without a new experiment.
Permanent holdouts as conflict avoidance: 5% left forever because nobody wanted to say kill.
No product owner present and a “provisional ship” that never gets confirmed.
Blaming the analyst for a true null. Nulls save engineering time if you accept them.

How this ties to the rest of AMS

Decision logs are a form of operational documentation, close in spirit to stewardship habits: who owned the call, what definition of success applied, what not to reuse. If your warehouse metrics disagree with the experiment platform, fix the contract before the next review (see data stewardship and quality themes on the site). If someone wants AI to “summarize the experiment,” treat that summary like any other analysis artifact: check the primary, guardrails, and decision against the source tables. For SQL trust patterns, see How to check AI-written SQL.

How to practice this week

Take one past experiment that never closed. Write the decision you should have logged in six lines.
Draft the 25-minute agenda into your team’s template doc. Remove one standing agenda item that never produced decisions.
Require a pre-read for the next readout. If it is missing, reschedule (painful once, useful forever).
Add a “parking lot” section so segment curiosity has a home that is not the decision.
After the next meeting, paste the decision log into the ticket within an hour. Measure yourself on that habit.

Quick recap

Experiment reviews decide. Result first, segments second (labeled), guardrails third, decision log last. Twenty-five minutes with the decision owner present beats an hour of charts without a verb. Ship, iterate, kill, or bounded extend. Everything else is notes.

That finishes Experimentation culture as a three-part arc: design beyond the p-value, peeking and stopping rules, and meetings that commit. Next series in the queue shifts to customer analytics: funnels that match the product, conversion math, and cohorts.

Sources

Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments: organizational process and decision hygiene: https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7ECBD9E84D
Georgiev, Statistical Methods in Online A/B Testing (practical decision framing): https://www.amazon.com/Statistical-Methods-Online-B-Testing/dp/1694079724
Atlassian team playbooks on decision records (lightweight ADR-style logging adaptable to experiments): https://www.atlassian.com/team-playbook/plays/decision
Microsoft EXP research group (industry experimentation platform practices): https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/
Google HEART framework (useful for choosing what “activation” and quality mean before the meeting): https://research.google/pubs/pub36299/