How CodeInstances Keeps the Failing Run of a Flaky CI Test

Here’s a situation most teams know. An integration test fails about one run in twenty. It calls an API that usually answers one way and occasionally answers another (a field missing, a list in a different order, a slow page that times out). Somebody hits rerun, it goes green, and the failure is gone. A week later it happens again, and the ticket still says “can’t reproduce”.

The trouble is that the one run worth looking at is the one nobody kept. Logs say the test failed. They don’t hold the answer that made it fail.

With CodeInstances you keep every CI run of that test as an instance. In a Go project, build the test binary first (so the recording holds the test’s traffic and nothing from the build), then put codeinstances record in front of it:

go test -c -o checkout.test ./integration
codeinstances record -o test-run.instance -- ./checkout.test -test.run TestCheckout

CI runs of the same test: most pass, one fails. The failing run is kept as failing.instance, a build artifact, and the pipeline still goes red because record exits with the test’s own code. Locally the instance is replayed offline as often as needed, the code is fixed, and the fixed build is replayed against the same file.

record exits with the program’s own exit code, so the step still fails when the test fails and the pipeline behaves exactly as before. The only change is that a file is left behind. Upload it as a build artifact when the step fails, along with the test binary (most CI systems can keep artifacts only on failure), and the passing runs cost you nothing.

Replay It Until the Fix Holds

Download the artifacts and replay the instance on your own machine, from a folder holding the test binary:

codeinstances replay test-run.instance

Replay never opens a connection. The test gets the same answers it got in CI, including the odd one, so it fails the same way, every time, offline. That’s the whole point. A one-in-twenty failure becomes a one-in-one failure you can step through with a debugger.

Once you’ve found the cause and fixed the code, replay the same instance against the new build:

go test -c -o checkout-fixed.test ./integration
codeinstances replay test-run.instance -- ./checkout-fixed.test -test.run TestCheckout

The fixed build sees exactly what the broken one saw. If it now passes, the replay reports what changed (the exit code and the test output lines), which is what you expect when a failing test starts passing. Keep the file. It’s now a record of the case that broke, and you can replay it after later changes to check the fix still holds.

Where It Bites

Two limits matter here. The test has to send its traffic through HTTP_PROXY and HTTPS_PROXY, which most clients do. And if the test is written in Go and talks to a server on localhost, Go’s client skips the proxy for that address, so point it at a host name and send that name to the local address with -resolve:

codeinstances record -resolve api.test=127.0.0.1:8080 -- go test ./integration

On macOS and Windows, Go ignores SSL_CERT_FILE, so a Go test there needs hook.Client() for HTTPS. On Linux CI runners it isn’t needed. If the flakiness comes from the clock or random numbers in Go code, the hook package records those too. In other languages only the HTTP side is captured, so a failure that depends on timing inside the program may not replay. Test runners also print timings, which differ on every run. A line like --- FAIL: TestCheckout (0.31s) will show up as an output difference on replay, so read the report for the request and exit code lines first. The reference has the rest of the limits.