How to write a verifier that fails

The verifier is a second agent that grades the first one. It exists because a model that produced work is the worst possible judge of that work: it will report the page is good, the claims are sourced, and the job is done, because that is what it was trying to produce.

The whole value is in whether it can fail. A verifier that has never returned FAIL has not been tested, and an untested verifier is worse than none, because it manufactures confidence.

Run it in a fresh session

This is not optional and it is the most common mistake.

If you ask the same session to check its own work, it sees its own reasoning. It knows why it made each choice, and it will grade the intent rather than the output. You get agreement, not verification.

Open a new session. Give it the spec and the output, and nothing about how the output came to be. It should be able to reach a verdict from the artifacts alone. This is the maker-checker pattern, and the separation is the mechanism.

Check facts, counts, and boundaries

A good verifier checks three kinds of things.

Facts. Does every number appear in the source it claims? Does every link resolve? Did an existing value change when it should not have?

Counts. Is every page in the input list present in the output? Are there as many sections as the brief required? Does every item that went in come back out?

Boundaries. Did the work touch anything outside its brief? Is there a file changed that should not have been? Did a drafts-only loop acquire a send path?

All three are mechanical. A stranger with the spec and the output reaches the same verdict you would.

Never check taste

The rule that matters: if the check requires an opinion, it does not belong in a verifier.

Do not ask whether the writing is clear, whether the argument is compelling, or whether the page is good. A model asked to judge quality will almost always approve, because quality judgments are soft and agreement is the path of least resistance. You will get a PASS that means nothing.

Taste is what the human gate is for. The verifier clears the mechanical failures so that when you read the work, you are spending your attention on the part only you can do.

Write it as failure conditions

Do not write “check that the page is correct.” Write the specific things that constitute failure, each one independently checkable.

FAIL if any number in the diff does not appear in <claims-file>.
FAIL if any existing number changed.
FAIL if any page in <list> is missing from the output.
FAIL if the diff touches a file outside <scope>.
FAIL if any source link returns anything other than 200.
Otherwise PASS. Print the verdict and, on FAIL, which condition failed and the
evidence for it.

Listing failures rather than successes does two things. It forces you to be specific, because a vague failure condition is obviously useless in a way a vague success condition is not. And it makes the output actionable: you learn which check failed and why, rather than a bare verdict.

Make it print evidence

A verdict with no evidence cannot be audited. Require the verifier to print what it found, not just its conclusion.

Bad: FAIL

Good: FAIL: claims check. The diff asserts "38 percent faster" and claims-and-proof.md has no entry for it. Line 41 of the diff.

The second one you can act on in ten seconds. The first sends you back to read everything.

Test it against a broken run

This is the step that separates a real verifier from a decorative one, and it takes five minutes.

Deliberately break the work. Take a good output and introduce exactly the failure your verifier is supposed to catch: change a number, delete a required section, add a claim with no source.

Run the verifier on it. It must return FAIL, and it must name the thing you broke.

If it passes, your verifier is broken. Not the work. Fix the verifier and try again.

Do this once per verifier, when you write it. Keep the broken version around, because when you change the loop later you will want to re-confirm the checker still catches it.

Our own checker failed this test the first time it ran: it was asked to confirm every claim carried a source URL, and it accepted a repo file path as a source. The fix was to require a resolvable URL. That failure is in the receipt, because a checker that never fails is not a checker.

What a verifier cannot do

It cannot tell you the work was worth doing. It cannot catch a plausible, well-sourced, entirely wrong argument. It cannot know that the whole premise was mistaken.

Those failures survive verification and reach your gate. That is the correct division: the verifier eliminates the mechanical errors so your judgment is spent on the ones that need it.

How to write a loop, the whole file the verifier lives in.
How to write a stopping condition.
Download a loop and read its verifier prompt, which is the fastest way to see the shape.