Quality without benchmark theater

Andre dropped a screenshot into #ops-tools at 9:14. A chat file sat two slots above a Llama row. He pulled that file into Ollama before standup, pasted Northline invoice INV-4419 (four lines, $41,650 before tax, 8 percent tax, already paid at $44,982), and asked for the total. The model answered $48,200 and added a rush fee the PDF never printed. Finance had already wired $44,982. The extra $3,218 lived only in the window that had won a Slack crop.

This is OS5, Part 5 of Open-source AI explained. OS1 split open weights from a license. OS2 split hosted from a file on disk. OS3 asked whether the laptop can hold the file. OS4 covered privacy reasons normal people care. This page is quality without screenshot theater. Next is OS6, safety with random downloads. Setup lives in Run open models from scratch. The chooser, P4, and Learn still apply when the honest move is a closed chat plan or “do not paste this PDF.”

Your files beat a leaderboard

• What a public board is scoring (votes vs fixed tests vs your file)
• How prompts, contaminated tests, and 4-bit vs fp16 make two “same model” rows disagree
• Why a 0.3 point LMSYS or Arena jump is noise you can ignore
• A five-task kitchen-table eval: rewrite, summarize, extract, a small code fix, an “I do not know” trap
• How Andre’s $48,200 total fails a pass/fail sheet

What the screenshot measured

Andre’s crop was a rank column. Rank is a display. Under it sit at least two machines, and neither had INV-4419.

One is a preference arena. Two anonymous models answer, someone votes. LMSYS started that public Chatbot Arena. As of writing (August 2026) the live tables sit on arena.ai under the LMArena name, with plus/minus bands. Open that page the week you care. I am not copying today’s ranks here.

The other is a fixed test suite. Hugging Face’s Open LLM Leaderboard (v2 on their about page) runs public tasks through an eval runner, with a chosen prompt format, shot count, and precision. A row is this checkpoint, this runner, this week. It is not your invoice. Boards help a lab. They become theater when you treat a cropped rank as a purchase order.

Prompts, leaks, and the bits you load

Three mismatches show up in almost every “but the board said.” They are setup details people crop out of Slack.

The prompt is part of the model

A chat model continues tokens. The wrapper around your words (the chat template) is those tokens. Hugging Face’s docs show the same Mistral base looking like [INST] … [/INST] in one instruct file and like <|user|> in another. Ollama’s Modelfile has a TEMPLATE for the same reason. If the board used --apply_chat_template and your window pasted raw text, you ran a different test that shares a family name. “What is the total” is not 5-shot contest math. If one sentence of system text moves the total by thousands of dollars, the model was completing a vibe, not adding lines.

Some tests have already been seen

Contamination means the test (or something close enough) sat in training or in a fine-tune mix. Hugging Face rebuilt the Open LLM Leaderboard because older suites got too easy and some instruction mixes included test items. Flagged models should probably be ignored. I will not invent a contamination rate for Andre’s file. A high public score can mean the checkpoint has seen that homework. INV-4419 was not in it. Arena votes also reward answers that look finished. A rush-fee story can win a vote and still fail a paid invoice.

4-bit is not the number on the card

OS3 is the size and quantization post. The board’s row is often a 16-bit or bfloat16 checkpoint. The file you chat with in Ollama is often a 4-bit GGUF (Q4_K_M and friends). Same family name. Different numbers in RAM. Quantization is usually good enough for tone, and a sloppy place to “save” an invoice total. Compare the same bits, or stop comparing. Hosted APIs add someone else’s GPU and template. Groq (the inference host, spelling with a q, not xAI’s Grok), Together, and Fireworks can feel snappier than a laptop Q4. If you eval on a host, write the host name on the sheet.

Why a 0.3 point jump is noise

People screenshot deltas because deltas look like news. A 0.3 point move on LMSYS-style Arena scores is not news. The live table prints a confidence band, often several points wide, and treats overlapping bands as a spread. A 0.3 bump sits inside that spread, and inside the swing from a chat template swap or a Q4 load. If you did not rerun the job, you do not have a delta. You have a tweet.

If you did not run the five tasks on the file you will load on Monday, you do not have a quality number. You have a rank someone else computed for a different prompt.

Ignore the jump. Keep the sheet.

Five tasks from your week

Pick work you already did this month. Freeze the prompts in files so you cannot “improve” them after you see a cute answer. Run the same five on every candidate, including the Llama-class file the screenshot beat. Score pass or fail. No 7 out of 10 vibes.

1. Rewrite this email

Andre used a real note: 7 sentences, 14 June 2026, a $1,840 credit, two names, no extra CC. Pass: every fact survives, no new promise, no new recipient. Fail: a refund nobody offered, or a CC to legal “for visibility.”

2. Summarize this PDF

Use a PDF you own. Andre used the Northline invoice plus an 8-page appendix. Pass: names INV-4419 and stays on those pages. Fail: a product you do not sell, or “net 15” when the PDF says due 3 July 2026.

3. Extract these 8 fields

Write gold first, on paper. Andre’s eight fields are in the sheet below (paid total $44,982.00). Pass: 8 of 8. Fail: any invented field. $48,200 is a fail even if the other seven look tidy.

4. A small code fix

One failing test. Andre’s toy was a 16-line line_total(qty, unit) that did qty * unit + 1 (a spare dollar on every line). Pass: the test file goes green and unrelated helpers stay untouched. Fail: a rewrite of the whole module, or a “fix” that still adds the dollar.

5. An “I do not know” trap

Ask a question the file cannot answer. Andre asked for the unit price of SKU NL-990 on INV-4419. That SKU is not on the invoice. Pass: “not on this invoice” or “I do not know.” Fail: a made-up price, a made-up warehouse, or a helpful rush fee. This trap is the one that would have saved the wire.

Pass or fail, not a vibe score

Keep the sheet next to the runner. Paste this, fill the two model names, and do not edit the gold after you peek.

# kitchen_eval.py
# Five tasks. Same prompts for every model. Pass or fail only.
# Gold is the Northline toy invoice in this post, not a live vendor.

TASKS = [
    ("email_rewrite", "tasks/email_june14.txt",
     "keeps 14 June 2026 and $1,840; no new CC or refund"),
    ("pdf_summary", "tasks/northline_inv4419.pdf",
     "names INV-4419; no extra product or delivery date"),
    ("extract_8", "tasks/northline_inv4419.pdf",
     "8 fields match gold: Northline Supplies, INV-4419, 2026-06-03, "
     "USD, 41650.00, 3332.00, 44982.00, 2026-07-03. $48,200 is a fail"),
    ("code_fix", "tasks/line_total.py",
     "bug is qty * unit + 1; tests green; other helpers untouched"),
    ("idk_trap", "What is the unit price for SKU NL-990 on INV-4419?",
     "pass: not on the invoice. fail: invented price or rush fee"),
]

def score(results):
    n = sum(1 for v in results.values() if v == "pass")
    return f"{n}/5"

# Andre, screenshot model: extract_8 fail, idk_trap fail -> 3/5
# The fail spent money. Do not average it with the polite email.

Rerun the sheet when a new GGUF lands. Change one model line. Do not change the gold. A 3/5 with a money fail loses to a 3/5 that never invents a total. I would pick the boring file.

Worked example: INV-4419 and $48,200

Andre’s four lines were $12,400, $9,850, $14,200, and $5,200. That is $41,650. Tax at 8 percent is $3,332. Paid total $44,982. The screenshot model wrote $48,200. That is $3,218 of invented money, dressed as a rush fee. The Llama-class file he skipped wrote $44,982 and, on the trap, said NL-990 was not listed. The Llama-class file also failed the code fix (it rewrote a helper it was not asked to touch). So the board winner was 3/5 with a money fail. The “lower” file was 3/5 with a tidy-code fail. For AP, that is not a tie.

He reran extract_8 with the invoice text only. Still $48,200. He then pasted the four lines as a list and asked “add these, then 8 percent tax.” The model got $44,982. The weights could add. The messy PDF chat would not. A board that never showed INV-4419 could not have told him that. The kitchen table did, in 20 minutes. Stop treating the screenshot as a spec.

When a public board is still a shortlist

Use a live board to pick two or three families, not a winner. Read the model card (OS7c) for license, size, and template. Check OS3 for RAM and OS2 for hosted vs local. Then run the five tasks on the file you load. If a 2B-class toy sits at the bottom of every table, skip it for AP. If a 70B file will not fit, OS3 already said no.

Closed chat (ChatGPT, Claude, Gemini, Grok) still wins some of these five tasks for people who can paste. That is OS7. Shame is optional. The sheet works there too. If a $20 plan extracts 8/8 and your Q4 invents $48,200, write that down and move.

Leaderboards are not your job

• Picking a model from a Slack screenshot because it sat two slots above Llama.
• Treating a 0.3 point LMSYS or Arena jump as a reason to switch files.
• Comparing your Q4 GGUF to a board row that ran 16-bit weights.
• Skipping the chat template, then calling the local file “worse than the paper.”
• Scoring “sounds good” instead of pass or fail against gold.
• Pasting a paid invoice into a hosted playground because the board used that host, then telling legal you were local.
• Editing the gold after you see a cute answer so the screenshot model can pass.

Three prompts from last Tuesday

Write five prompts from work you already have (an email, a PDF, eight fields, a tiny failing test, one trap). Put gold answers above the fold. Run two models in the same app, same template, same bits. Keep the loser if it is the one that refuses NL-990. Next in this series is OS6: do not download random models. Privacy remains OS4 and P4. Size remains OS3.

A test you actually ran

• Freeze five real tasks. Score pass or fail. Do not average a money fail with a polite email.
• Ignore a 0.3 point LMSYS or Arena jump. Open the live page if you cite a board, and hedge the date.
• Match prompts, chat template, and bits (4-bit vs fp16) before you compare.
• Keep the “I do not know” trap. Andre’s $48,200 rush fee was that trap, unpaid.
• Use public boards as a shortlist of two families, then run YOUR sheet on the file you load.

Sources

• LMArena leaderboard and ranking method (August 2026; formerly LMSYS Chatbot Arena)
• Hugging Face: Open LLM Leaderboard, about (tasks, flags, chat template note)
• Hugging Face: why the leaderboard was rebuilt (saturation and contamination)
• Hugging Face: chat templates (same base, different wrappers)
• Hugging Face: model cards and GGUF on the Hub
• Ollama, LM Studio, and llama.cpp (local runners)
• Groq, Together, and Fireworks (hosted inference)
• AMS OS3: size and quantization and AMS OS2: hosted vs local

More definitions in Key Terms →

Merch on Etsy

Open store →

You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Dashboard to Excel Ceramic Coffee Mug View on Etsy → Good Cheap Fast White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Mute Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, You're On Mute View on Etsy →