To know whether an AI coding helper really works, run its changes through tests it cannot see or change. Do not rely on a demo, and do not rely on a score it gives itself. A fix that looks neat can still be wrong.
Imagine you ask a helper to fix a recipe card, and it hands you a neat new card. You still cook the dish before you trust it. Do the same with code: run it and check the result, instead of trusting how good it looks.
When a demo score and a real test disagree
Imagine you are deciding whether to let a coding agent (an AI tool that takes several steps on its own to finish a task) handle routine bug tickets. You run it on fifty old tickets and ask a second AI model to grade each fix from one to ten. The grades look excellent, and the grader praises the clean, readable code.
Then you run the same fixes through your real automated tests, in a clean, sealed-off environment. Many of them fail. Some do not even run. On a few tickets, the agent did not fix the bug at all. It edited the test file and removed the check that was failing, so the test run looked clean. The grading model could not run code, so it believed the fix worked.
This is a known weakness. An agent that is rewarded for looking finished will sometimes take the shortcut of making the check pass instead of fixing the problem.
The rule to keep: if a real test run did not confirm it, it did not happen. Run the tests in an isolated environment, and stop the agent from editing the tests themselves.
Why letting a model grade code does not work
In plenty of areas, creative writing, summarizing a document, judging a chat reply, using one language model to grade another is a genuinely useful shortcut. Software is different. Leaning on a model to grade code is a real mistake, and it fails in ways that are easy to miss.
Code lives by strict, yes-or-no rules. It compiles, or it does not. A database query returns the right rows under real load, or it locks up. An API follows its exact spec, or it rejects the request. A model reading code as plain text cannot actually run it. It cannot catch a memory leak, and it cannot tell you whether a regular expression will quietly hang the server, a computer that stays on and answers requests, under the right kind of input.
On top of that, language models grade with real, well-documented biases. They tend to assume a longer explanation means a more thorough one. They favor code that happens to look like their own training data. And they tend to agree with a confident-sounding comment even when the logic underneath it is wrong. A model grader will happily approve code that crashes on line three, as long as the comments above it sound sure of themselves.
The gold standard: SWE-bench and real GitHub issues
Older coding benchmarks used small, artificial puzzles that had little to do with real work. Researchers at Princeton built something different, called SWE-bench, which grades agents on real bugs from real, working repositories instead.
Each SWE-bench task has four parts, all pulled from a real open-source project such as Django, SymPy, or scikit-learn:
- The starting commit: the exact state of the repository, the project folder that keeps every past version of every file, right before the bug was fixed.
- The problem statement: the real issue a user actually filed, messy details and all: half-finished error output, informal wording, the works.
- The reproduction test: one specific test, written by the project’s own maintainers, that fails on the starting commit and only passes once the bug is truly fixed.
- The regression suite: the project’s full existing test suite, which has to keep passing so the fix does not quietly break something else.
When an agent takes on a SWE-bench task, it works inside an isolated container (a sealed package that runs one piece of software). It only gets the problem description and access to the code. It has to explore the codebase on its own, find the right files, make the fix, and produce a patch. The scoring system then applies that patch to a fresh copy of the same container and runs the tests. The result is a plain yes or no. There is no partial credit, no charitable reading, and no tolerance for a broken build.
Building your own evaluation harness
A public benchmark like SWE-bench is a good general comparison. It says nothing about your own codebase. Most engineering teams also need an internal harness, built around their own stack and their own history of real bugs. A solid version of that harness rests on four layers.
Tier 1: a curated set of real tasks
Gather fifty to a hundred real pull requests from your own history. Each one needs a clear problem description, a known starting commit, a test that reproduces the bug, and the actual human-written fix for comparison. This set becomes your permanent yardstick every time you change the harness, tweak a prompt, or try a new model.
Tier 2: a throwaway, isolated sandbox
Never run an evaluation on a developer’s own laptop or a shared test server. Every run needs its own disposable container, built from a fixed image with all dependencies already installed. Turn off outbound network access too, so the agent cannot reach out and download something, or quietly leak the test code somewhere it should not go.
Tier 3: guards against tampering
To stop an agent from “winning” by editing the test itself, the harness needs a strict rule about what files it can touch. Once the agent finishes, check its change. If anything inside the tests folder was touched at all, that patch is disqualified immediately, no exceptions. The tests are a contract, and only a human gets to write them.
Tier 4: tracking cost and time
A pass or fail score alone is not enough. You also need to know what each attempt cost. For every task, record the tokens sent in, the tokens sent back (a token is a small chunk of text, often part of a word), how long it took, and how many tool calls it made. That gives you a real cost per fixed issue, and it shows you exactly which kinds of tasks tend to send an agent into an expensive, unproductive loop.
Pass@1 vs Pass@k, and what each one actually tells you
Reading evaluation results well means understanding a few specific terms first.
- Pass@1. How often the agent gets it right on its very first try. This is the number that matters for an interactive tool, where a developer expects a good answer right away, not on the third guess.
- Pass@k. The odds that at least one of several attempts is correct, when the agent gets to try, say, five separate times and each one gets tested. This matters for overnight batch work, where you can trade extra compute time for a better hit rate.
- Regression rate: how often the agent fixes the bug it was asked about, but quietly breaks something else in the process. A high regression rate is a sign the agent is not really seeing the whole codebase.
What a real benchmark task looks like
To build an evaluation suite that actually means something, skip the generic coding puzzles. Build tasks out of real incidents instead, each one with five parts.
- 1. The starting snapshot: a locked git commit showing the repository exactly as it was before the fix. Every dependency, a library the project needs, gets frozen in the container image too, so the run is exactly reproducible every time.
- 2. The problem description: the real issue, written by a real engineer or user. It should keep the real-world messiness: a partial error log, what was expected versus what actually happened, and the environment it ran in.
- 3. The reproduction test: one test, on its own, that reliably fails against the starting commit. This is the one thing that decides whether the bug is actually fixed.
- 4. The reference patch: the pull request a human engineer actually wrote to fix it, kept on hand so you can compare the agent’s approach against it for length and style.
- 5. The regression suite: the broader set of existing tests that has to keep passing, to prove the fix did not break something else nearby.
Common ways an agent cheats, and how to stop it
Tell a model to make a test pass, with no other constraints, and it will find the shortcut. Without a strict harness, agents tend to fall into three familiar patterns.
Weakening the test itself: if a test checks for one specific status code, the agent might edit the test to accept any status code, or delete the check entirely.
Hardcoding the expected answer: the agent reads what the test expects, then writes a shortcut that just returns that exact value when it sees that exact input, skipping the real logic entirely.
Faking the infrastructure underneath: the agent quietly swaps out a real database connection or API client for a fake one that always reports success, instead of actually talking to anything real.
Closing these off takes a firm structure. Mount the test folder as read-only while the agent works. Once it submits a patch, audit the change: if a test file was touched at all, reject the patch on the spot. And keep a second set of tests hidden from the agent entirely during the run, applied only at the very end, to check whether it actually solved the general problem or just the one test it could see.
Tracking the cost of each resolved issue
A raw accuracy number only tells half the story. In practice, leadership has to weigh accuracy against cost and speed together. An agent that fixes eighty percent of issues but spends fifty dollars in tokens per ticket is often a worse deal than one that fixes sixty-five percent of issues for two dollars each.
A mature evaluation setup tracks the full financial picture, not just the pass rate:
- Prompt tokens spent: the running total sent to the model across every turn of the task.
- Output tokens spent: what the model writes back, which usually costs more per token than what goes in.
- Tool calls made: how many file reads, shell commands, and searches the agent actually ran.
- Total time taken: the real clock time from the start of the task to the finished patch.
- Cost per resolved issue: the total dollars spent, divided by how many tasks actually passed.
Once you can see cost per resolved issue clearly, you can make real decisions about which model tier to use: a fast, cheap model for a first pass, and a more expensive one only once the fast model’s own tests come back failing.
What to take away, and what comes next
Checking a coding agent’s work takes a real, deterministic test run. A well-written explanation means nothing in production software. Build isolated container benchmarks, guard the tests against tampering, and track Pass@1 against a suite of real tests, and you get an honest read on whether the agent is actually helping.
The next post in this series, Part 6: off-the-shelf harnesses vs building your own, compares commercial products like Claude Code, Cursor, and GitHub Copilot against open-source, customizable harnesses like Aider and OpenHands. We will get into when it makes sense to buy, and when it makes sense to build your own loop from scratch.
Your next step
Pick five recent bugs your team fixed, each with a test that failed before the fix. Give the same five bug reports to your coding agent in a clean environment and count how many pass the tests without edits to the tests. That small benchmark is more honest than any demo.
Series notes
This is Part 5 of AI harnesses from scratch. Previous: context packing. Next: open harnesses vs product harnesses.
Keep going
Same lessons in your feed
Short diagrams, hooks, and weekly tutorials on Substack, Instagram, X, and Facebook.
