Inclusive metrics and whose story is missing

The dashboard says activation is up. Leadership smiles. Product ships more of whatever moved the line. Three months later support tickets climb among users who never got a fair chance to activate: no payment method yet, shared household accounts, regions with flaky SMS, people who only use the product at work on locked-down devices. The metric was not “wrong” in SQL. It was incomplete as a story about who succeeds.

This is Part 2 of Inclusive data products. Part 1 covered charts real users can read. This part goes one layer deeper: inclusive metrics, or noticing whose experience never enters the numerator, denominator, or segment cut before the number becomes a decision.

What you’ll learn

Why metric inclusion is a product and ethics problem, not only a SQL filter
A four-step loop: metric → who is in → who is out → action
How proxies (zip code, device, language) can smuggle harm into “neutral” KPIs
A short inclusion review you can attach to metric specs
How to practice finding missing stories without freezing every launch

Inclusive metrics start with who can appear

Every metric is a camera angle. Cameras always crop. Inclusive analytics does not pretend the crop is the whole world. It makes the crop explicit, checks who falls outside the frame, and decides whether the decision still holds.

Common crops analysts inherit without noticing:

Logged-in users only when many real customers browse logged out
Completed accounts when the painful drop happens before account creation
English UI events when localization ships later
App-only behavior when a large segment uses the web or partner channels
Paid customers when free users are the pipeline you claim to care about
US-centric time windows when global usage peaks overnight for HQ

None of these crops is automatically evil. Measuring paid retention for a finance forecast is fine. Trouble starts when a cropped metric is used to claim “customers are happy” or “the product works for everyone,” or when optimization loops punish the invisible group.

Rule of thumb: If you cannot name who is excluded from a KPI, you are not measuring a population. You are measuring convenience.

The loop: metric, who is in, who is out, action

Use a simple four-box walk every time a metric graduates from “exploratory” to “steering.”

Filled missing-story flow: metric, who is in, who is out, action

1. Metric: name the decision, not only the formula

Write one sentence: “We will use X to decide Y.” If you cannot fill Y, you do not need a production metric yet. Inclusive work fails when teams debate denominators without knowing the decision. A churn definition for board reporting and a churn definition for win-back campaigns can both be valid and still need different inclusion rules.

Tie this to metric discipline from the metrics series: owner, grain, numerator, denominator, filters, and refresh. Inclusion is another required field on that card, not a separate manifesto.

2. Who is in: the population that can score

List the eligibility rules in plain language:

What event or status puts someone in the denominator?
What time window must they survive before counting?
Which product surfaces must they use?
Which countries, plans, or segments are in scope?

Then reverse the sentences into people. “Accounts with at least one successful payment method and a completed onboarding checklist by day 7” is a formula. In human terms it may mean “people who already cleared identity, banking, and UI hurdles we do not measure.”

3. Who is out: the missing story

Ask deliberately cruel questions:

Who tries and never becomes eligible?
Who is active but untracked (shared logins, offline, partner portal)?
Who is filtered out as “noise” (small regions, low volume languages, new platforms)?
Who appears only as a proxy (zip, device model, ISP) that correlates with protected or sensitive traits?
Who would look like a failure because our instrumentation is biased, not because they failed?

Write the outs as named groups, not as residual “other.” “Users without a verified phone number in markets where SMS is unreliable” is actionable. “Outliers” is a way to stop thinking.

4. Action: what changes because of the gap

Inclusion without action is journaling. Choose at least one:

Widen the metric (include more surfaces or pre-account steps)
Add a companion metric (funnel step for the excluded group)
Segment the steering metric so averages cannot hide harm
Stop using the metric for a decision it cannot support
Fix instrumentation so the missing group can appear fairly
Change product or ops when the metric revealed a real barrier

Proxies, fairness, and “neutral” features

Analysts often get pulled into scoring, ranking, or prioritization models: lead scores, credit-adjacent limits, support priority, fraud flags, content moderation queues. Even when you never train a neural net, a simple SQL tiering rule is a decision system.

Proxies are attributes that stand in for something you cannot or should not measure directly. Zip code as income. Device price as “serious user.” Nighttime usage as “bot.” Language as “support cost.” Some proxies are useful. Many encode history we would not defend out loud.

Inclusive metric practice for proxies:

Name the proxy and the intended construct (“zip as rough income for marketing mix,” not “zip as creditworthiness”)
Ask who is systematically mis-ranked if the proxy is wrong
Prefer outcome metrics you can defend (paid invoice, verified delivery) over lifestyle correlates
When legal or policy teams care (they should), escalate early; analytics silence is not neutrality

You do not need to become a fairness researcher overnight. You do need to refuse “the model said so” as a complete sentence when the inputs were zip, phone OS, and browser language.

Worked example: activation rate and the people who never count

Scenario: a B2C app defines activation as “created account, completed profile photo, and completed first project within 7 days.” Weekly activation rate is the north-star for growth. The rate rises after a campaign that targets users already familiar with similar tools.

Walk the loop.

Metric / decision: Use 7-day activation to decide whether onboarding experiments ship. Grain: user account. Window: first 7 days after signup.

Who is in: Users who successfully create an account and hit client-side events for photo upload and project create. Requires modern browser eventing, photo upload bandwidth, and a project template that assumes desktop-sized screens.

Who is out (examples):

People stuck before account creation (email verification failures, school or work domain blocks)
Users on low bandwidth who skip photo upload and never “activate” by definition
Shared household devices where one account serves three people (instrumentation undercounts humans)
Languages where project templates are incomplete, so “first project” is harder for non-reasons
Users who complete meaningful work via a partner embed that does not fire the same events

Action options the team actually ships:

Add a companion metric: % of signups that fail email verification by domain type
Make photo optional in the activation definition for low-bandwidth markets, or track “core project” without photo
Segment activation by language and device class so the average cannot hide a regression
Instrument partner embed events so those users can appear in the same story

Here is a compact inclusion review table you can paste into a metric one-pager.

Inclusion review table asking who cannot appear, proxy harm, and decision impact with examples

Filled for this activation metric:

Ask	Answer for activation
Who cannot appear?	Pre-account failures; partner-embed users; blocked email domains
Proxy harm?	Photo requirement proxies bandwidth and device quality
Decision impact?	Onboarding experiments optimize for already-resourced users
Companion metric?	Verification success rate; activation by language and device
Stop using for?	Claims that “the product works for all new users”

A tiny SQL sketch (toy names) shows how easy it is to hide people in a WHERE clause that looks professional:

-- Steering metric as currently defined
SELECT
  DATE_TRUNC('week', u.signed_up_at) AS signup_week,
  COUNT(*) FILTER (
    WHERE u.photo_uploaded_at IS NOT NULL
      AND u.first_project_at <= u.signed_up_at + INTERVAL '7 days'
  )::float / NULLIF(COUNT(*), 0) AS activation_rate
FROM users u
WHERE u.account_status = 'active'
  AND u.signup_surface = 'main_app'  -- partner embed excluded
  AND u.locale IN ('en-US', 'en-GB') -- "for now"
GROUP BY 1
ORDER BY 1;

The filters may be temporary. Temporary filters have a habit of becoming “how we measure success.” Inclusive practice is to put those filters on the metric card in human language and schedule a revisit date.

Segments, small n, and the ethics of averages

Teams sometimes avoid cutting metrics by language, region, disability-related assistive tech flags, or other slices because volumes are small or privacy is sensitive. Both concerns are real. Neither justifies forever averages.

Small n: Use longer windows, hierarchical summaries, or qualitative research instead of declaring the group irrelevant
Privacy: Aggregate, suppress cells under a threshold, and avoid publishing re-identifying cross-tabs; talk to privacy counsel for sensitive attributes
Missing attributes: If you lack a field, do not invent it with a proxy; measure the process barrier instead (error rates, completion time, support contacts)
Quality of the cut: A bad self-reported field can create false confidence; treat data quality of demographic or access fields as seriously as revenue fields (data quality series)

Inclusive metrics often look like better product analytics: funnels that start earlier, instrumentation that covers all surfaces, and guardrail metrics next to the north star. The moral language and the craft language agree more than people expect.

How this connects to charts and stewardship

Part 1’s accessible chart rules still apply. A beautifully labeled line of a biased metric is still a biased story, only easier to read. Conversely, inclusive metric definitions still need inclusive presentation: if only one team can decode the caveats buried in a hover, the missing story stays missing in the meeting.

Stewardship also matters. Who may see which slices, how long you retain sensitive attributes, and what “official” means are governance questions. Pair this post with your team’s data stewardship habits and the broader path on data stewardship when access and ownership get real. For a learning map of related skills, use Learn.

Common mistakes

Optimizing a cropped KPI while claiming universal success. Scope the claim to the population.
Calling excluded users “noise.” Noise is instrumentation failure or fraud. People are not noise.
Proxy features without naming the construct. Zip is not income. Device is not intent.
One average to rule the roadmap. Guardrails and segments prevent silent regressions.
Inclusion theater. A paragraph in a doc with no companion metric or product change is not a review.
Freezing all launches until perfect fairness. Perfect is a stall tactic. Ship improvements and measure who still cannot appear.
Leaving inclusion to “the ethics committee” only. Analysts write the WHERE clauses. Own them.
Ignoring support and sales qualitative signal. Tickets often name the missing story before the warehouse does.

How to practice this week

Pick one steering metric you report in a recurring meeting.
Write the decision it is supposed to drive in one sentence.
List eligibility rules, then rewrite them as “who is in” in human language.
Name three groups who cannot appear or who appear unfairly. If you cannot name three, ask support or sales for candidates.
Fill the inclusion review table (who cannot appear, proxy harm, decision impact).
Propose one action: companion metric, segment, instrumentation fix, or narrowed claim language.
Add the inclusion note to the metric’s definition doc so it survives the next owner.

This series stops at two parts on purpose: presentation (Part 1) and definition (Part 2). Together they are a minimum bar for inclusive data products. The next series in the queue shifts to career skills: how analysts grow as ICs, build portfolios, and earn senior trust with clear storytelling.

Quick recap

Every metric crops reality; inclusion means naming the crop and the people outside it.
Walk metric → who is in → who is out → action before a KPI steers product or policy.
Proxies can smuggle harm into “neutral” scores; name the construct and the failure mode.
Companion metrics and segments beat a single flattering average.
Accessible charts (Part 1) and inclusive definitions (Part 2) need each other.
Small, scheduled improvements beat perfection stalls and empty ethics paragraphs.

Sources

National Institute of Standards and Technology (NIST). AI Risk Management Framework (useful language for measuring and managing system impacts, including socio-technical risk). https://www.nist.gov/itl/ai-risk-management-framework
UK Government Statistical Service. “Harmonisation” and inclusive data guidance (practical public-sector framing of who is counted). https://analysisfunction.civilservice.gov.uk/policy-store/inclusive-data-taskforce-recommendations-report-and-implementation-plan/
Office for National Statistics (UK). Inclusive Data Taskforce materials and related updates. https://www.ons.gov.uk/aboutus/whatwedo/programmesandprojects/inclusivedata
Google PAIR. People + AI Guidebook (human-centered questions around datasets and evaluation, transferable to KPI design). https://pair.withgoogle.com/guidebook
Data Feminism (D’Ignazio and Klein). Book site and open chapters on power, counting, and whose knowledge counts. https://data-feminism.mitpress.mit.edu/
W3C WAI. Accessibility fundamentals (reminder that exclusion is not only visual). https://www.w3.org/WAI/fundamentals/accessibility-intro/