The Agent Said “Done.” What Did It Actually Prove?

The change was small, the logic looked reasonable, and the tests covered the exact cases the agent said were missing. I could have sent it for review and moved on. Instead, I stopped at one question:

What did those passing tests actually prove?

They proved that the new code handled two example internal addresses used to route login requests. They didn't prove that the application could ever receive those addresses. The installed login library used different, longer addresses that the original code already covered.

It was like testing a new lock without first confirming that the building had the door it was made for. The lock could work perfectly and still prove nothing about whether that door existed.

Every visible signal looked positive: the code changed, the tests passed, and the agent said it was done. But moving it forward could still waste review time on a fix for a problem no one had shown to exist.

Before deeper review, I wanted a narrower answer: what had been confirmed, what was still missing, and what should happen next.

I started with the exact version of the login library installed in the project: Better Auth 1.6.2. Its sign-in and sign-up routes included:

A small classifier decides which login requests receive the stricter rate limit. The original code already matched those longer addresses. The agent added two shorter ones:

The new unit tests showed that both shorter addresses now selected the login rate limit. That result was useful, but narrower than the agent's claim. Better Auth 1.6.2 didn't define either address as a login operation, so the tests didn't show that a real request could reach the new code.

To confirm the claimed fix, I needed to see a real request use one of those addresses, or find version-matched documentation or configuration showing that it existed. Because this was login protection, I would also need to send repeated requests and see the application allow the early ones and block the request over the limit. The new tests did neither.

Without those checks, I couldn't say, “The bug is fixed.” I could only say:

The code handles two more inputs. I haven't shown that the application uses them or that the login protection works for them.

At this point, it's fair to ask: isn't this what code review is for? Not quite.

Code review looks at the implementation. It asks whether the logic is correct, safe, maintainable, and a good fit for the system.

But good code can still solve a problem no one has shown to exist.

The missing answer here wasn't about code quality. It was whether the tests matched the application's real behavior. Agent verification asks that first: did the agent complete the task a person had accepted, and do the results support what it reported?

I need both verification and code review, but they don't answer the same question. Checking the report first can also save review time by showing the reviewer what is already confirmed and where to look first.

Instead of accepting or rejecting the whole report, I split it into smaller claims. Each one needed a result that could actually support it:

This didn't turn the whole change into a failure. It kept the useful part: the code handled two new inputs, and the tests for them passed. It also exposed the missing link: nothing showed that those inputs occurred in the application.

I also checked that the green result belonged to the same code I was reading. A test can pass, the code can change, and the final report can still say, “Tests pass.” The result is real, but it belongs to older code.

In this case, the result matched the change. I didn't need a long audit trail, only the repository, the exact version of the code, the command, and its outcome. The skill calls that exact version the repository snapshot.

There was one more limit. The only description of the task came from the agent's own report. It told me what the agent believed it was fixing, but not what a person had approved before the work began.

The skill records that limit under Contract Authority. Because the task description was agent-derived, the highest possible verdict was PARTIALLY_VERIFIED.

In plain language, the code change and its unit tests were confirmed. The reported bug fix was not. My verification verdict was:

That verdict kept the confirmed work without pretending the main claim was proven.

The next action was separate from that verdict:

HOLD didn't mean rejecting the code. It meant not calling it a verified bug fix until I could show that the application used one of those addresses and that the login protection behaved correctly. The other option was to describe the change more narrowly as defensive handling for inputs that hadn't been shown to occur.

The right check depends on the claim. The rule is simple: the check must be able to answer the question being asked.

A unit test can show that a function returns the expected result for chosen inputs. It can't show that a button works in a browser, that a payment changed someone's access, or that one customer can't see another customer's data.

This doesn't mean every change needs every check. It means choosing the smallest check that can answer the real question.

The cost of being wrong decides how far I go. A copy change may need only a readback and link check. A normal bug fix needs a test of the changed behavior. Login rules, payments, database changes, and agent actions need stronger proof because a wrong conclusion can affect access, money, data, or an external system.

The depth of verification should match the consequence of being wrong. That keeps small tasks small without lowering the bar for risky changes.

But what happens when the right proof isn't available?

The limit on the verdict came down to one unanswered question:

Did a person ask the agent to fix this specific problem, or did the agent define the problem itself?

The report placed that question under Cannot Verify and recorded exactly what was missing: the original prompt, accepted criteria, or another human-approved task source. Any of those could show whether a person had approved this diagnosis before the agent implemented it.

Cannot Verify wasn't a second verdict. It was a place to record an unanswered question and whether it blocked the next step. It didn't mean the agent had failed or that the code should be thrown away.

Here, the missing task source limited the verdict. It did not erase the code change or the passing tests.

I turned this check into a public Codex skill on GitHub called aga-verify-agent. I use it after a coding agent reports that its work is complete and before I decide where the work should go next.

The check follows a simple path:

It doesn't replace code review or testing the actual behavior, and it doesn't approve a merge. It shows what is supported, what is still missing, and where the work should go next.

Light, Standard, and High-Risk modes keep the depth proportional to the change. After installing the skill, run $aga-verify-agent. The repository includes installation steps and examples.

So what did “done” prove in this case?

It proved that the code handled two additional inputs and that their unit tests passed for the code I reviewed. It didn't prove that the application used those inputs or that the change fixed a real gap.

The next step was to hold the bug-fix claim until a real request or version-matched source established that one of those inputs could occur, then test the login protection itself. Otherwise, the change needed a narrower description.

That was enough to make the decision. I didn't need to trust the whole report, and I didn't need to throw all of the work away.