Review, tests, and not trusting green checkmarks

Friday, 4:47 p.m. Codex just finished a “small” fix: the dashboard filter was ignoring archived accounts. The chat summary is cheerful. A row of green checkmarks sits in the terminal like a participation trophy. You feel the weekend. Your finger hovers over push. Then you open the diff. Forty-three files. A deleted test suite. A new constant named API_KEY with a real-looking string. The original bug fix is somewhere in the middle of a reformat that rewrote half the utils folder. You close the laptop for a second, not because the model is evil, but because shipping blind is how small fixes become Monday incident tickets.

This is Part 4 of the ChatGPT Codex / coding tutorial. Part 3 covered skills, tasks, and scheduled help as product offers you climb into carefully. This part is the brake that makes those power-ups safe: how to read the diff, run the tests yourself, hunt secrets, check scope, and keep a human on the commit. Green checkmarks help. They do not prove the change is what you asked for, or safe for production. If you still need Chat vs Work vs Codex orientation, use the ChatGPT product map. Everyday verification habits sit in the ChatGPT everyday tutorial. Work’s multi-step result checks live in the ChatGPT Work tutorial. Foundations stay in Learn ChatGPT from scratch.

What you’ll learn

A five-step review loop after every agent change set
How to read a diff like a skeptical teammate, not a grateful passenger
Why “tests passed” is a rumor until you run them (or see real CI)
A secrets pass that catches keys, tokens, and fake-real placeholders
Scope checks that stop “helpful” monorepo rewrites
A diff smell checklist and a worked recovery for a bloated “small fix”

Permission UIs and default modes move. Treat git commands below as portable patterns. Re-check your team policy and current Codex docs before you write org-wide rules.

Why green checkmarks are not a review

Codex is good at producing plausible change sets. That is the product. Plausible is not the same as correct, complete, or scoped. A change can:

Pass a unit test that never covered the real edge case
Fix the bug and also “clean up” three unrelated modules you did not ask for
Delete a flaky test instead of fixing the flake
Hardcode a secret “just for local” that later lands on a shared branch
Rename something globally in a way that breaks a plugin you forgot existed

If you only look at the chat summary, you review the story the agent told about the work. If you look at the diff, you review the work. Those are different jobs. Analysts already know the pattern from AI-written SQL: fluent narrative, wrong grain. Same class of risk with code. For the SQL version of this habit on AMS, see How to check AI-written SQL before you ship it.

Green checkmarks are especially seductive because they look objective. A passing suite is evidence about the cases the suite covers. It is not evidence that the product intent is right, that no secret slipped in, or that the agent stayed inside the ticket. Treat CI like a second opinion, not a notary seal.

The review loop (do not skip steps)

After Codex finishes a multi-file change, run this loop in order. Make it boring. Boring is the point.

1. Read the diff

Start with the file list, not the first hunk. Ask: does this list match the ticket? Open each file that is outside the obvious blast radius and ask why it is there. Then read the hunks that touch business logic, auth, money, PII, migrations, and config. Skim pure formatting only after you know nothing dangerous hides inside a “style” commit.

Useful git views (run them yourself; do not only trust the agent’s paraphrase):

# What changed?
git status

# File list + short stats
git diff --stat

# Full unstaged patch
git diff

# Staged only
git diff --cached

# Against main (adjust branch name)
git diff main...HEAD

In an IDE, the same idea is a side-by-side diff for every touched file. Terminal or GUI is fine. The non-negotiable is that a human eyes the actual patch before the commit leaves your machine (or at least before it hits the shared default branch).

2. Run the tests that matter

“The agent said tests passed” is a rumor until you run them in your shell (or see CI on the PR). Prefer the project’s documented commands from README, AGENTS.md, package scripts, or your team wiki. If the project has a fast suite and a slow suite, run the fast suite always and the slow suite when the change touches risk paths (auth, payments, data migrations, shared libraries).

# Examples only. Use your repo’s real scripts.
npm test
npm run typecheck
npm run lint

# Or
pytest -q
go test ./...
cargo test

If tests fail, fix or roll back. Do not “commit green later.” If tests pass but you never understood them, you still have not reviewed the logic. Tests reduce risk. They do not replace reading the auth change.

Also watch for the anti-pattern where the suite got greener because coverage shrank. If the stats show deleted test files, assume guilt until proven innocent.

3. Check for secrets

Search the diff for keys, tokens, connection strings, private URLs, and customer data samples. Agents sometimes invent placeholders that look real, or paste from local env files into source. Both are bad. Placeholders that look real confuse the next human. Real secrets in git are an incident.

# Crude but useful smell search on the patch
git diff | rg -i 'api[_-]?key|secret|password|token|BEGIN (RSA |OPENSSH )?PRIVATE|AKIA[0-9A-Z]{16}'

# Also check newly added files
git status --short

If something secret landed in the working tree, remove it before commit. If it already committed to a shared branch, follow your security process (rotate the secret, scrub history only with people who know what they are doing). Do not casually rewrite public history alone at 5 p.m.

4. Check scope

Scope is the question: “Did we only do the job we asked for?” Agents love helpful extras: rename for consistency, extract a helper, update docs, reformat imports, upgrade a dependency “while we’re here.” Helpful extras can be good on a dedicated cleanup PR. On a bugfix PR they hide the real change and expand the blast radius of review.

A practical rule: if the ticket was “fix filter on archived accounts,” the PR should mostly be the filter, tests for the filter, and maybe a one-line comment. If the PR is also a TypeScript upgrade, that is two tickets wearing one jacket.

5. Human commit

You (or a teammate with ownership) write the commit message and decide what lands. Let Codex draft a message if you want, then edit it until it is true. “Assorted improvements” is not a message. “Fix archived-account filter in dashboard query; add regression test” is a message.

git add path/to/relevant/files
git commit -m "Fix archived-account filter in dashboard query"

# Push a feature branch, not necessarily main
git push -u origin fix/archived-filter

Default habit: feature branch + pull request. Direct push to main is a team policy decision, not a model permission prompt. Part 5 covers git-friendly beginner habits in more depth. For this part, assume you still own the commit button.

Rule of thumb: Chat summary is a pitch. Diff is evidence. Tests are a second opinion. Commit is a signature.

Diff smells: stop and reassess

Use this checklist when something feels “off,” or as a routine gate for larger agent runs. Any one item is enough to pause. Two items means you almost certainly need to split, revert, or re-prompt with a tighter goal.

Smell	What it often means	What to do
Unrelated files changed	Scope creep or wrong root cause	Restore those files; re-ask for a narrower edit
Deleted tests without a clear reason	Agent “fixed” red by removing proof	Restore tests; fix the code or mark skip with a ticket
Hardcoded secrets or keys	Env confusion or bad example code	Remove, rotate if needed, use env/secret store
Huge rewrite for a tiny bug	Overfitting to a vague prompt	Reset; restate the bug with file/line constraints
“Just trust me” commit message	Nobody can review intent later	Rewrite message; if you cannot, you do not understand the patch
New dependencies you did not ask for	Convenience over control	Demand justification; check license and supply chain
Permission or auth code touched “by accident”	High blast radius	Slow down; second reviewer; extra tests
Force-push suggested	History rewrite as cleanup	Refuse on shared branches; prefer revert

When you find a smell, say it out loud to the agent in the next turn: “Revert changes outside src/filters/ and the related test. Do not reformat. Do not touch package.json.” Tight constraints after a bad run work better than scolding the model for being creative.

Undo without making it worse

Undo is a skill. The wrong undo turns a local mess into a team mess. Learn the safe defaults first.

Discard uncommitted edits (usually safest)

If Codex edited files and you have not committed, you can throw away changes per file or for the whole tree.

# Throw away unstaged edits to one file (modern git)
git restore path/to/file.ts

# Throw away all unstaged edits in the working tree
git restore .

# Unstage without discarding contents
git restore --staged path/to/file.ts

# Older equivalent people still use
git checkout -- path/to/file.ts

git restore is the clearer modern command for “make the working tree match HEAD (or another source).” Prefer it when your git version supports it. Official reference: git-restore.

Undo a local commit carefully

If you committed only on your machine and have not pushed (or only pushed to a personal branch you fully control), you can move the branch pointer.

# Soft: undo commit, keep all changes staged
git reset --soft HEAD~1

# Mixed (default): undo commit, keep changes unstaged
git reset HEAD~1

# Hard: undo commit AND discard those changes (destructive)
git reset --hard HEAD~1

--hard is a shredder. Use it when you are sure you want the agent’s work gone. Prefer soft or mixed when you still want to salvage pieces. Official reference: git-reset.

Force-push is dangerous

Force-push rewrites history on the remote. On a shared branch it can erase a teammate’s commits, break open PRs, and confuse CI. Even on a personal feature branch it can surprise anyone who already pulled. Treat git push --force and git push --force-with-lease as tools for known, deliberate recovery, not as the default cleanup after an agent mistake.

If Codex suggests force-pushing to “clean up” a bad commit on main, that is a red flag. Stop. Prefer a reverse commit, a new fix commit, or a controlled reset only if your team’s process explicitly allows it and you understand who else has the branch.

# Safer pattern for a bad commit already on a shared branch:
# add a reverse commit instead of rewriting history
git revert HEAD
git push

Worked example: the “small filter fix”

Imagine you asked:

In src/dashboard/query.ts, include archived accounts only when
includeArchived is true. Add a unit test. Do not reformat other files.
Do not change dependencies.

Codex returns a cheerful “done.” You run the review loop.

$ git diff --stat
 src/dashboard/query.ts           |  12 ++++--
 src/dashboard/query.test.ts      |  28 +++++++++++++
 src/utils/format.ts              | 140 ++++++++++++++++++----------------
 src/utils/dates.ts               |  88 +++++++++++---------
 package.json                     |   2 +-
 tests/legacy/dashboard.spec.js   | 210 --------------------------------
 .env.example                     |   1 +
 7 files changed, 200 insertions(+), 281 deletions(-)

What the stats tell you before you even open hunks:

query.ts and query.test.ts look on-mission.
format.ts and dates.ts look like unrelated rewrites.
package.json was not requested.
A large test deletion is a classic smell.
.env.example might be fine or might hide a bad pattern; open it.

Recovery path:

# Keep the good files, drop the rest of the agent’s surprise
git restore src/utils/format.ts src/utils/dates.ts package.json tests/legacy/dashboard.spec.js

# Inspect remaining diff carefully
git diff

# Run tests
npm test -- src/dashboard/query.test.ts

# Commit only the intentional paths
git add src/dashboard/query.ts src/dashboard/query.test.ts
git commit -m "Respect includeArchived in dashboard query"

Then tell Codex what you did and why, so the next turn does not re-apply the junk:

I restored format.ts, dates.ts, package.json, and the legacy dashboard spec.
Keep only query.ts + query.test.ts for this task. Do not reformat utils.
Do not delete tests. Confirm with git diff --stat before more edits.

That conversation pattern matters as much as the git commands. Agents will happily re-expand scope if you only delete files quietly and keep chatting as if the wider plan still stands.

Permissions help, review still wins

Codex and the ChatGPT desktop coding surfaces have approval flows, sandboxes, and plan-gated capabilities that change over time. Those rails are real protection. They are not a substitute for reading the patch. Approving “write to these three files” still requires you to understand what was written. High-trust modes that skip prompts raise speed and risk together.

If Part 3 left you tempted to schedule multi-file jobs while you sleep, this part is the answer: schedule only when the worst failure is a draft, and still run the review loop before anything merges. Autonomy without a merge gate is cosplay.

A 15-minute practice drill

Do this on a throwaway branch this week, even if you already “know git.”

Ask Codex for a deliberately underspecified change in a sample repo (“improve error messages a bit”).
Stop when it claims done. Run git diff --stat only. Write three smells you can see without opening hunks.
Open the full diff. Mark one on-mission hunk and one off-mission hunk.
Restore the off-mission files. Re-prompt with the Part 3 task template bounds.
Run real tests. Commit only the intentional paths with a true message.

The point is muscle memory under low stakes, so Friday at 4:47 feels automatic instead of heroic.

Where this sits in the ChatGPT path

Learn ChatGPT from scratch
ChatGPT product map
ChatGPT everyday tutorial
ChatGPT Work tutorial
ChatGPT Codex / coding tutorial (this series: open → explore → skills/tasks/schedule → review → git habits next)
Sibling brake pedal on Anthropic’s stack: Claude Code tutorial (review and undo patterns transfer)

Next in this series: git-friendly habits for beginners (branches, small commits, PR culture, never force-push blind). After Codex, the ChatGPT track moves into Custom GPTs for repeating non-repo playbooks.

Quick recap

Loop: read diff → run tests → check secrets → check scope → human commit.
Green checkmarks are a second opinion, not a ship button.
Diff smells: unrelated files, deleted tests, secrets, huge rewrites, vague messages, surprise deps, force-push talk.
Prefer git restore and honest reverts over history rewrites on shared branches.
Tell the agent what you restored so the next turn does not re-bloat the PR.

Sources

Research and further reading used for this article:

OpenAI: Codex product page (coding agent product context)
OpenAI: Introducing the Codex app (agent workflows and review-oriented surfaces evolve; re-check current UI)
Git: git-diff (inspecting patches and stats)
Git: git-restore (discarding or unstaging safely)
Git: git-reset (moving branch pointers carefully)
Git: git-revert (undoing published commits without rewriting history)
AMS: How to check AI-written SQL before you ship it (same “fluent but wrong” habit in SQL)
AMS: ChatGPT Codex / coding tutorial (this series)
AMS: ChatGPT Work tutorial (result checks for office agents)
AMS: Learn (full series index)