Choosing a model for work (not hype)

Your team opens a spreadsheet of model names, pastes last week’s “arena” rankings into Slack, and asks which one should power customer support summaries, SQL drafts, and the new internal bot. Someone has a favorite. Someone else has a budget. Nobody has written down what “good” means when the answer is wrong at 4pm on a close day.

Model choice for work is not a beauty contest. It is a product decision under constraints: latency, cost, data policy, eval quality, and how badly a confident wrong answer hurts. Leaderboards measure something. They rarely measure your workflow.

This guide is a one-shot playbook for analysts, analytics engineers, and team leads who need a default model stack without chasing every release note. If vocabulary still feels fuzzy, start with What are LLMs, ChatGPT, generative AI, and more. If the model will write SQL, pair this with How to check AI-written SQL. For broader hands-on AI habits, see the Practical AI series on Learn paths at Analytics Made Simple Learn.

What you will learn

Why public leaderboards are a weak default for work systems
The real axes: task fit, latency, cost, context, tools, privacy, and failure cost
A simple decision matrix you can fill in one meeting
A worked scorecard for three common data-team jobs
Mistakes teams make when they “just pick the smartest one”
A practice drill that produces a written model policy, not a vibe

Leaderboards measure a different game

Public comparisons are useful the way a restaurant rating is useful: they tell you something about average taste under someone else’s menu. They do not tell you whether the kitchen can handle your rush hour, your allergens, or your budget for 10,000 meals a day.

Common leaderboard blind spots for analytics work:

Task mismatch. Elo on open-ended chat is not Elo on “rewrite this dbt test without changing grain.”
Prompt and tooling mismatch. Your system prompt, retrieval, and tool schema change outcomes more than a half-point rank swap.
Latency and cost ignore. A “smarter” model that takes 12 seconds and costs 8x can lose to a mid-tier model with a tighter prompt.
Data residency and logging. The winner may not be allowed to see customer rows under your policy.
Drift. Model versions change. Yesterday’s champion is tomorrow’s silent behavior shift.

Use public scores as a shortlist filter, not a purchase order. Your shortlist should still lose to your own eval set when the two disagree.

The work axes that actually matter

Write these down before anyone says a model name. If you cannot fill the table, you are not ready to choose a model. You are ready to choose a pilot scope.

1. Task shape

Separate jobs even if they share a chat UI:

Drafting: SQL, Python, email, docs (human edits expected).
Classification / extraction: label tickets, pull fields from text, map free text to taxonomy.
Summarization: meeting notes, long tickets, incident timelines.
Q&A over private knowledge: “what does metric X mean?” with retrieval.
Agents with tools: call warehouse, ticket system, or code runner.

A model that is great at long prose may be mediocre at strict JSON. A model that is great at code may over-explain and ignore your refuse rules. Task shape drives eval design.

2. Latency and interactivity

Interactive IDE assist wants sub-second to a few seconds. Batch overnight enrichment can tolerate minutes. Mixed products fail when one slow model path sits behind a “snappy” UI promise. Measure p50 and p95 on your real prompt sizes, not demo prompts of three sentences.

3. Cost at your volume

Price lists are per million tokens. Your bill is tokens times volume times retries times tools. A cheap model that needs three repair turns can cost more than a stronger model that gets structure right once. Include human time: minutes of analyst cleanup are real money even when the invoice is small.

4. Context, tools, and multimodality

Do you need large context for long schemas? Tool calling that is reliable under your JSON schema? Image input for chart screenshots? Streaming? Structured outputs? Pick the capability floor first. Everything below the floor is not a candidate, no matter the rank.

5. Policy: privacy, retention, region

Ask legal and security early, in writing: training on prompts? Zero data retention options? Regional processing? What PII is banned from prompts? What must stay in-VPC or on approved vendors only? A model that cannot legally see the data is not “almost fine.” It is disqualified.

6. Failure cost

Wrong product description draft: low. Wrong patient summary or wrong finance total in a board pack: high. High failure cost demands stronger evals, human gates, and often a conservative model plus verification, not maximum creativity.

Decision matrix for work models: task latency cost policy failure cost versus leaderboard rank

The diagram is the meeting artifact. Fill cells with constraints, not opinions. If two models both clear the floor, then you run a bake-off on your eval set.

Build a tiny eval before you fall in love

You do not need a research lab. You need 20 to 50 frozen examples that look like production pain:

Real (or realistic sanitized) prompts
Expected structure (SQL that runs, JSON keys, summary length, refuse behavior)
Hard cases: ambiguous grain, missing columns, conflicting instructions, empty result sets
Pass criteria a second person can grade without heroics

Score each model the same way. Track automatic checks (parseable JSON, query executes, latency) and human checks (correct grain, no invented columns, useful tone). Re-run when models or prompts change. Version the set like any other test fixture.

For SQL-heavy work, your eval should include “did it invent a column” and “does the join match the business grain,” not only “does it look SQL-ish.” That is where AI SQL checking becomes policy, not preference.

Worked example: three jobs, one scorecard

Imagine a mid-size analytics team with three near-term uses:

Job A: IDE assist for analysts drafting SQL and small Python transforms.
Job B: Nightly batch: classify support tickets into a fixed taxonomy and extract product names.
Job C: Internal Q&A bot over metric definitions and runbooks (retrieval allowed, no raw customer tables in context).

Constraints the team already agreed:

No customer PII in prompts without approved redaction.
Interactive assist should feel under about 5 seconds p95 for typical prompts.
Batch can spend more per item if accuracy is higher and human review is lower.
Bot answers must cite retrieved snippets; inventing policy text is a fail.

They shortlist three model tiers (names abstracted so this stays about process, not a sales pitch):

Fast-cheap: low latency, low cost, weaker on hard reasoning.
Mid: solid code and structure, moderate cost.
Heavy: strongest reasoning, highest cost and latency.

After a one-week bake-off on 40 frozen cases per job, the scorecard looks like this (toy numbers for teaching; replace with yours):

Job	Primary pick	Why	Backup
A IDE SQL/Python	Mid	Best pass rate on join/grain cases; p95 latency acceptable	Fast-cheap for tiny renames
B Ticket classify	Fast-cheap	F1 within 2 points of Heavy at 1/6 cost; structured JSON stable	Mid for low-confidence bin
C Metric Q&A	Mid + retrieval	Heavy not worth latency; refuse-without-cite improved with better chunks	Heavy only for hard multi-doc disputes

Filled model scorecard for IDE assist batch classify and metric Q and A

Notice what the scorecard did not do: it did not crown one model for the whole company. Routing is a feature. Many mature stacks use a cheap model first, then escalate hard cases. That is operations, not indecision.

Sample scorecard fields you can copy

job: ticket_classify_v1
candidates: [fast-cheap, mid, heavy]
n_eval: 40
metrics:
  json_valid: 0.98 / 0.99 / 0.99
  label_f1: 0.81 / 0.83 / 0.84
  p95_latency_ms: 900 / 2200 / 6100
  cost_per_1k_items_usd: 0.40 / 1.10 / 3.80
policy_ok: yes for all (redacted text only)
decision: fast-cheap default; mid if conf < 0.6
owner: analytics-platform
review_date: +90 days

Store that next to the prompt and retrieval config. When someone asks “why not switch to the new shiny model,” you point at the card and the eval set, not at a screenshot of a public chart.

Routing, not monogamy

Sensible patterns for data teams:

Default + escalate: cheap model drafts; mid or heavy only on validation failure or low confidence.
Task map: different defaults for SQL, prose, and classification.
Offline vs online: batch can afford heavier models; interactive paths stay lean.
Human gate on high stakes: model proposes; person ships finance or customer-facing numbers.

Routing needs observability: which model ran, which prompt version, latency, and whether the human overrode. Without logs, you cannot improve the scorecard.

Policy is part of the architecture

Treat model access like warehouse access. Who can send production schemas? Who can enable tools that run queries? Where do prompts land in logs? How long are they retained? Which vendors are approved?

Practical minimums:

Written allowlist of use cases and data classes.
Redaction or synthetic substitutes for demos and training material (see also synthetic data thinking elsewhere on this site’s AI and quality writing).
No secret keys in notebooks; use managed secrets.
A kill switch: disable a model path without redeploying the whole company.

Risk frameworks such as the NIST AI Risk Management Framework are not a model picker, but they are a reminder that “it worked in a demo” is not risk acceptance. Document residual risk the same way you would for a flaky pipeline that still runs in production.

Common mistakes

One model to rule them all. Different jobs have different floors and failure costs.
Leaderboard shopping without evals. You optimize for someone else’s benchmark.
Ignoring p95 latency. Averages hide the turn that kills the meeting demo.
Counting only API dollars. Human cleanup and rework dominate true cost.
Skipping policy until after launch. Retrofitting retention and PII rules is expensive and awkward.
Never re-testing after model upgrades. Silent regressions are real; pin versions when you can and re-run the golden set.
Confusing “feels smart” with “passes structure.” Pretty prose that invents columns is a fail for data work.
No owner. Someone must own the scorecard, the eval set, and the review date.

Practice: run a one-week model decision

Pick one real job this week (not three). Write the constraint table: task, latency target, monthly volume estimate, data classes, failure cost. Build 20 frozen examples with pass criteria. Shortlist two or three candidates that clear policy. Run the same suite. Fill the scorecard. Write a three-line decision: primary, backup, escalate rule. Put a 90-day review on the calendar.

If you cannot get 20 examples, you do not understand the job yet. Shadow the people who do the work. Steal their messy prompts. Those are the eval gold, not textbook exercises.

When SQL is in scope, add at least five cases that should refuse or ask for clarification when columns are missing. Pair the habit with the SQL checking tutorial linked above so verification stays human-owned.

Quick recap

Leaderboards shortlist; your evals and constraints decide.
Axes that matter: task shape, latency, cost at volume, capabilities, policy, failure cost.
Build a small frozen eval before you fall in love with a model name.
Route by job: default, backup, escalate. One company-wide “best model” is usually a myth.
Write the scorecard, own it, re-run on upgrades, and treat policy as architecture.
For data work, structure and honesty beat vibes every time.

Sources

OpenAI, models and pricing documentation (capabilities and token pricing patterns change; verify current pages): https://platform.openai.com/docs/models
Anthropic, Claude models overview: https://platform.claude.com/docs/en/about-claude/models/overview
Google, Gemini models documentation: https://ai.google.dev/gemini-api/docs/models
NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0): https://www.nist.gov/itl/ai-risk-management-framework
LMSYS Chatbot Arena (public preference leaderboards; use as shortlist signal only): https://chat.lmsys.org/
Analytics Made Simple, Practical AI series: https://analyticsmadesimple.com/series/practical-ai/
Analytics Made Simple, How to check AI-written SQL: https://analyticsmadesimple.com/tutorials/how-to-check-ai-written-sql/