Testing AI-generated code against the spec

When a coding agent writes the code and its tests in one go, a green test run tells you the code agrees with the tests, not that the tests were strong. We asked Claude Code five times to write a semantic-version comparison and its tests. All five implementations passed an independent test suite we wrote from the specification, and each agent's own tests caught six of seven bugs we injected. The seventh, a comparison that ignores letter case, was missed by all five. This is one small task and one model, so read it as a method and a first data point.

Lab note · who it is for: developers who accept code and tests from a coding agent · observed October 22, 2026 · Claude Code 2.1.306 (model claude-sonnet-5-5), Node.js v24.7.0, macOS

The question

An agent that writes a function also writes the tests for it, runs them, and reports success. Three things can be wrong at once: the function, the tests, and your assumption that the two are independent. They are not: both come from the same reading of the task. The check that does not share that reading is a test suite written from the specification, and a way to measure how much of the specification the agent's own tests actually cover.

The setup

• The task is comparing two semantic version strings. We chose it because the specification (Semantic Versioning 2.0.0, items 10 and 11) is short, public and precise, and it has a few rules that are easy to get wrong. It is also very common, which probably helps the model.
• The prompt was one sentence and did not mention any rule. It was run in a new Git repository each time, with the permission mode set to acceptEdits and only node commands allowed.

Implement compare(a, b) in semver.js (CommonJS, module.exports = { compare }). It compares two semantic version strings and returns -1, 0 or 1. Also write tests in semver.test.js using node:test, and run them with node --test.

• The oracle is a small test suite we wrote from the specification. We confirmed it passes our reference implementation and fails each of the seven bugs below.
• The bugs are seven versions of the reference implementation with one rule broken each. We chose them from the specification's rules, so they are not a random sample and not exhaustive.

// oracle.test.js: written from semver.org 2.0.0, items 10 and 11, not from any model's output
const chain = ["1.0.0-alpha", "1.0.0-alpha.1", "1.0.0-alpha.beta", "1.0.0-beta",
               "1.0.0-beta.2", "1.0.0-beta.11", "1.0.0-rc.1", "1.0.0"];
// every pair in the chain compares in order
compare("1.2.10", "1.2.9") === 1                      // core is numeric
compare("1.0.0+a", "1.0.0+b") === 0                   // build metadata is ignored
compare("1.0.0-alpha.2", "1.0.0-alpha.11") === -1     // numeric identifiers compare as numbers
compare("1.0.0-1", "1.0.0-alpha") === -1              // numeric ranks below alphanumeric
compare("1.0.0-alpha", "1.0.0-alpha.1") === -1        // more fields rank higher
compare("1.0.0-Z", "1.0.0-a") === -1                  // ASCII order: uppercase before lowercase

numeric-as-string          pre-release numbers compared as text ("11" < "2")
numeric-above-alpha        a numeric identifier ranks above an alphanumeric one
core-as-string             major, minor, patch compared as text ("10" < "9")
case-insensitive           identifiers compared ignoring case
build-compared             build metadata takes part in the comparison
prerelease-above-release   1.0.0-rc.1 ranks above 1.0.0
longer-not-higher          a longer pre-release field set is not higher when the prefix is equal

What we measured

For each of the five runs we checked three things. First, whether the agent's implementation passes the oracle. Second, whether the agent's own tests pass its implementation. Third, how many of the seven bugs the agent's own tests catch. For the third check we ran the agent's test file against the reference implementation and against each bug, and counted a bug as caught when a test that passes on the reference fails on it.

What this shows, and what it does not:

• The code was right. On this task, the code the agent wrote matched the specification as we tested it. The risk we set out to measure did not show up in the implementation.
• The tests had the same blind spot every time. None of the five test files compared two identifiers that differ only in letter case, such as 1.0.0-Z and 1.0.0-a. A change that makes the comparison case-insensitive would pass all five suites. The specification says identifiers compare in ASCII order, which puts uppercase first.
• The agent's tests also asserted things the specification does not say. Between one and three tests per run failed on our reference implementation because they check choices the agent made: invalid input throws a TypeError, and a leading v is accepted. Those are design decisions, not spec requirements. We ignored them in the count above, but you should notice them: the specification's FAQ says that v1.2.3 is not a semantic version. One run also tested very large numeric identifiers; our reference compares them as JavaScript numbers and fails that test, so that failure is a limit of our reference, not of the agent's code.

What to do with this

1. Write the oracle before you read the agent's tests. Derive it from the specification or from cases you already know are right. A dozen lines per rule is enough.
2. Break the code on purpose and see what turns red. A handful of injected bugs, one per rule, shows which rules the agent's tests never exercise. It takes minutes and does not need a mutation-testing tool.
3. Read the tests for decisions. If a test asserts how invalid input is handled or which spellings are accepted, ask whether you chose that.
4. Ask for the gap. After the first pass, ask the agent which rules its tests do not cover. We did not test whether it finds the case-sensitivity gap; that is an experiment worth running.

What we did not verify

• Codex. We tried five runs with Codex, and the account's usage limit stopped every one before any file was written. There is no Codex result here.
• Any other model, a longer or less well-known task, or code that has to deal with state, I/O or concurrency.
• Whether a more detailed prompt, or a request for edge-case tests, closes the gap. We used one sentence on purpose.
• Whether the oracle is complete. It tests the rules we chose; a bug in a rule we did not think of would pass.
• Five runs give a rough picture, not a rate.

Independence

Nothing here is sponsored, and no vendor saw this note before it was published. We use Claude Code ourselves, so we are not neutral about it. The results are limited to the runs described.

Reproduce it

Create five new Git repositories and run the prompt in each. Write the oracle from the specification. For the bugs, copy your reference implementation five or six times and break one rule in each copy. Run each agent's test file with node --test against the reference and against each broken copy; a bug counts as caught when a test that passes on the reference fails on it. Everything uses Node.js built-ins.

References

• Semantic Versioning 2.0.0 (read October 22, 2026), items 10 and 11
• Node.js documentation: Test runner
• Treat model output as untrusted input, for code that handles what a model returns