The analysis PR checklist

Software teams learned the hard way that “it works on my machine” is not a release process. Analytics teams still ship board metrics with a Slack shrug: “looks good to me.” Then two quarters later nobody can explain why churn changed definition in March, or why a join quietly doubled revenue for one region.

This post is a one-shot guide to the analysis PR checklist: a practical review gate for SQL, metrics, and decision memos whether you use GitHub, GitLab, or a careful doc plus diff habit. You will get a review flow, a checklist you can paste into pull request templates, and a worked example of a bad change caught before merge. No series membership required.

What you’ll learn

What an “analysis PR” is even if your company does not use that phrase
A review flow from draft question to merge and communication
Checklist items that catch real failures: grain, filters, fan-out, time logic, tests, narrative
How to review as a peer without becoming a blocker theater department
A worked example and a 20-minute practice for your next metric change

What counts as an analysis PR

An analysis PR is any proposed change that alters how a number is defined, computed, visualized for decisions, or interpreted in a deliverable others will reuse. That includes:

SQL for a certified metric or dashboard dataset
dbt models or transformation code that feeds reporting
Python that produces a recurring executive number
A notebook “final” export that leadership will quote
A definition change in a metrics catalog or semantic layer

Ad-hoc exploration can stay loose. The moment the artifact becomes shared truth, treat it like code: reviewable, testable, reversible. That mindset pairs with metric definition work in the metrics series and quality habits in the data quality series.

The review flow

A good analysis review is not “read every line once.” It is a short path with clear exits.

Analysis PR flow from question draft through checks peer review merge and communicate

Stages in plain English:

Frame: author states question, audience, and decision horizon.
Define: grain, inclusions, exclusions, and owner of the definition.
Implement: code or config change with readable structure.
Self-check: author runs the checklist before requesting review.
Peer review: second brain looks for failure modes, not style nits only.
Merge: change lands with version note or changelog line.
Communicate: consumers hear what changed and whether history backfills.

Skip communication and you will still get “the dashboard is wrong” tickets when the dashboard is merely different on purpose.

The checklist (copy this)

Use the following as a PR template section. Authors check boxes. Reviewers sample the risky ones deeply.

Analysis PR checklist covering question grain filters joins time tests and narrative

1. Question and scope

What decision does this support?
Who is the primary consumer?
Is this exploratory, recurring, or certified?
What is explicitly out of scope?

2. Grain and entities

One row means ___ (finish the sentence).
Primary keys are stated and uniqueness is checked.
Aggregations match the grain (no accidental person-level averages mixed into account-level sums).

3. Filters and business rules

Every filter is justified in the PR text (test users, refunds, status codes).
CASE logic matches the written definition.
Null handling is intentional, not accidental dropouts.

4. Joins and fan-out

Join keys listed.
Row counts before and after each risky join.
Many-to-many risks called out; dedupe strategy if needed.

5. Time logic

Timezone and week-start conventions stated.
Event time vs processing time distinguished if both exist.
Late data and backfill behavior described.
Period comparisons use aligned windows.

6. Lineage and inputs

Upstream tables or extracts listed.
Freshness expectations noted.
If AI drafted SQL or code, verification steps are listed (see how to check AI-written SQL).

7. Tests and validation

At least one automated or scripted check for uniqueness, nulls, or ranges.
Spot-check against a known period or a second method.
Impact preview: old vs new metric for last N periods.

8. Narrative and consumer impact

PR description explains the change in business words first.
Dashboard titles, metric descriptions, or catalog entries updated.
Communication plan for breaking changes.
Rollback path exists (revert commit, feature flag, or dual-run period).

Author habits that make reviews fast

Reviewers are not mind readers. Help them.

Keep PRs small enough to finish in one sitting when possible.
Put the business summary above the code dump.
Include a small results table: old vs new for the last 4-8 periods.
Link the definition doc or paste the grain sentence.
Call out the scariest join yourself. Reviewers will still check it, but trust rises.

Example impact table in a PR body:

Week	Old net revenue	New net revenue	Delta %
2026-W10	1,240,000	1,183,000	-4.6%
2026-W11	1,301,000	1,244,000	-4.4%
2026-W12	1,278,000	1,219,000	-4.6%
2026-W13	1,335,000	1,271,000	-4.8%

A table like that turns vague anxiety into a concrete consumer conversation: “Finance, we are removing unpaid invoices from net revenue. Expect ~5% lower weekly figures going forward.”

Reviewer habits that catch real bugs

Style nits are optional. These are not:

Read the definition before the SQL. If they disagree, stop.
Hunt fan-out: any join to a table that is not obviously unique on the key.
Hunt silent filters: WHERE status = 'paid' buried in a CTE.
Hunt time zone and week boundaries near executive metrics.
Ask: “What would make this number look good while being wrong?”

If you are the only analyst, still do a self-review the next morning, or trade reviews with a friendly engineer. Solo is not an excuse for zero friction. It is a reason to keep the checklist short and non-negotiable.

What “approve” should mean

An approval is not a social favor. It is a claim that the reviewer believes the change will not silently corrupt shared numbers under the failure modes they checked. That claim has limits. Reviewers cannot guarantee the entire warehouse. They can guarantee they read the definition, inspected risky joins, and saw an impact preview that matches the story in the PR.

Write that expectation down once for the team. When someone rubber-stamps, they are not being “nice.” They are transferring risk to every consumer who trusts the metric. When someone blocks over a color in a chart, they are burning review capital. Keep the bar high on correctness and proportionately loose on cosmetics unless the cosmetic confuses the decision.

For certified metrics, require at least one reviewer who did not author the change. For exploratory work that will never ship to a shared dashboard, skip the ceremony. The checklist is a scalpel, not a blanket. Misusing it on throwaway analysis creates process allergy, and then people avoid it when board numbers are on the line.

Worked example: the “improved” revenue join

Author submits a PR: “Fix revenue by joining invoice line items for more detail.” Reviewer opens the diff.

Old sketch (invoice header grain):

SELECT
  DATE_TRUNC('week', i.invoice_date) AS week,
  SUM(i.amount) AS revenue
FROM invoices i
WHERE i.status = 'paid'
GROUP BY 1;

New sketch (line items join without care):

SELECT
  DATE_TRUNC('week', i.invoice_date) AS week,
  SUM(i.amount) AS revenue
FROM invoices i
JOIN invoice_lines l ON l.invoice_id = i.invoice_id
WHERE i.status = 'paid'
GROUP BY 1;

Checklist failures:

Grain: sum still uses header i.amount while joining lines, so multi-line invoices multiply revenue.
Join fan-out: no before/after row counts in the PR.
Impact table: missing; would have shown a sudden jump.
Definition: PR title says “fix” but actually changes the metric identity if lines were meant to sum l.line_amount instead.

A corrected approach either stays at header grain or sums line amounts explicitly:

SELECT
  DATE_TRUNC('week', i.invoice_date) AS week,
  SUM(l.line_amount) AS revenue
FROM invoices i
JOIN invoice_lines l ON l.invoice_id = i.invoice_id
WHERE i.status = 'paid'
GROUP BY 1;

-- self-check idea
-- SELECT invoice_id, COUNT(*) FROM invoice_lines GROUP BY 1 HAVING COUNT(*) > 1;

And the PR must include old vs new only after the logic is coherent. Otherwise you are comparing two wrongs or a wrong to a right without a story.

Sample self-check queries authors can paste into the PR:

-- uniqueness of invoice header
SELECT invoice_id, COUNT(*) AS n
FROM invoices
GROUP BY 1
HAVING COUNT(*) > 1;

-- fan-out risk
SELECT
  COUNT(*) AS invoice_rows,
  (SELECT COUNT(*) FROM invoices i JOIN invoice_lines l ON l.invoice_id = i.invoice_id) AS joined_rows
FROM invoices;

If joined_rows is much larger than invoice_rows, any sum of header fields after the join is a red alert.

Lightweight tooling without bureaucracy

You do not need a 40-page process. Minimum viable analysis PR culture:

Git (or equivalent) for metric SQL and transform code
A PR template with the checklist section
One required reviewer for certified metrics (can be a tech lead or analytics engineer)
A changelog or release note channel
Optional CI: SQL lint, dbt tests, or simple uniqueness checks

Pipeline context helps reviewers know where the model sits. If you are wiring scheduled transforms, the data pipelines series is useful background. For general skill paths, see Learn.

Common mistakes

Review as theater: approvals without reading joins.
Style-only comments: renaming variables while missing fan-out.
Huge omnibus PRs: twelve metrics at once so nobody can reason.
No impact preview: consumers learn in the executive meeting.
Silent definition edits: code changes without catalog or dashboard text updates.
Tests that only assert “rows > 0”: useless against sophisticated wrongness.
Skipping communication: merge is not the end of the analysis change.

Practice: 20 minutes on your next change

Pick one metric query you already maintain. Create a fake PR description in a doc:

Write the grain sentence.
List filters and why each exists.
Identify the riskiest join; write a row-count check.
Build a 4-period old vs new table (even if new equals old today).
Write one consumer message you would send if the definition changed by 5%.

If you cannot complete those five steps, the metric is not review-ready even if the SQL “runs.”

Quick recap

Shared numbers deserve PR discipline: frame, define, implement, check, review, merge, communicate.
Checklist focus: question, grain, filters, joins, time, lineage, tests, narrative.
Authors provide impact tables and scariest-join callouts.
Reviewers hunt silent wrongness, not only formatting.
Small process, real checks, explicit consumer communication.

Sources

GitHub Docs, about pull requests and review culture: https://docs.github.com/en/pull-requests
dbt Labs, testing and documentation guidance for analytics engineering: https://docs.getdbt.com/docs/build/data-tests
Google SRE Book, chapters on meaningful monitoring and release discipline (concepts transfer to metric changes): https://sre.google/sre-book/table-of-contents/
PostgreSQL documentation (join behavior and aggregation reference): https://www.postgresql.org/docs/current/
The Turing Way, peer review and reproducible collaboration practices: https://the-turing-way.netlify.app/
ISO/IEC 25010 software quality model overview (quality characteristics useful when framing review criteria): https://iso25000.com/index.php/en/iso-25000-standards/iso-25010