CodeInstances Shrinks a Failing Run to the Answer That Breaks It and Keeps It as a Test

CodeInstances v0.3.0 adds two verbs at once. codeinstances shrink takes a failing run and a passing one and finds which of the failing run’s answers actually cause the failure. codeinstances test replays a folder of instances as regression tests. They belong together: the first finds the bug’s smallest form, the second makes sure it stays fixed.

Here’s the problem shrink goes after. You’ve got a run that fails and one that doesn’t, and diff from the last release shows you how they differ. Often that’s dozens of answers: timestamps, ids, counters, prices that moved. One of them broke the program, and the rest is noise. Reading them one by one is slow, and it’s guesswork.

So shrink doesn’t read them, it tries them. It replays the program offline with the passing run’s answers swapped in for some of the failing ones, sees whether it still fails, and narrows down from there. This is delta debugging, the technique Andreas Zeller described for finding the parts of an input that matter. It stops when no single remaining answer can be swapped out without the failure going away.

What Shrink Gives You

A line for each answer the failure needs, and a file. The line says which call it was and how the answer differed, the same way diff does: a status of 500 where the passing run got 200, or the JSON member that changed. The file, shrunk.instance, is the failing run with every answer it doesn’t need taken from the passing run, and only the calls the program made. Replay it and the program fails with the same exit code.

A run counts as the same failure when it ends the same way: the same exit code, or the same signal. When “exit 1” is too broad, -match adds a regular expression the output has to match too, so a different error that happens to exit 1 doesn’t count.

It’s honest about what it can’t do. Only answers to calls both runs made can be swapped. If the cause is a call that only the failing run made, shrink says the program fails with the passing run’s answers too, and tells you how many such calls there are. It also refuses when the failure doesn’t happen again on replay. A program that fails only some of the time can mislead it, since every run is taken at its word. -max-runs caps how many times the program runs, and Ctrl-C stops the search cleanly.

Instances as Regression Tests

codeinstances test testdata/ replays every instance in the folder. A run that passed when it was recorded has to replay exactly, so a change that alters what the program sends or prints gets caught. A run that failed when it was recorded has to exit 0 now, against the same answers that once broke it. That’s a regression test for the bug, built from the real failure and not from a hand-written mock.

It exits 0 when every test passes and 1 when any fails, so it drops into CI as it is. A shrunk failure makes a good test, since it holds only what the bug needs.

Tested

77 Go tests pass, and the race detector finds nothing. The self-checking demo has 18 checks now. It records a third run while its HTTPS server breaks one item, then shrinks that failing run against the first. 22 answers differed between the two runs. Shrink found the one that matters, item 7’s, in 10 runs and 339 ms. Then the demo puts the first run and the shrunk failure in a folder: test fails the program as it is and passes a fixed build that carries on when an item can’t be read. An independent review before release found a run limit that wasn’t a hard limit and an interrupted run that was counted as a result. Both are fixed. The result files are in test/results.

Next on the plan is grouping: telling when many failing runs are the same bug. Download v0.3.0, and the reference has every flag.