How to Upgrade a Dependency With CodeInstances Replaying Real Runs

You need to bump something underneath your program. A new major version of the HTTP client library, a vendor’s SDK, or the Go toolchain itself. The changelog is long. Your unit tests pass, but they use mocks written two years ago against the old behaviour, so passing doesn’t tell you much. What you’d like to know is simple: does the program still do the same thing on real work?

Real runs answer that. Before the upgrade, record a handful of runs of the current build doing real work, against the real services:

codeinstances record -o runs/report-daily.instance -- ./app report --daily
codeinstances record -o runs/report-monthly.instance -- ./app report --monthly
codeinstances record -o runs/sync-full.instance -- ./app sync --full

Pick runs that cover the paths you care about. Each one keeps every request the program made, every answer, what it printed and how it exited.

Record before the upgrade, replay after it. Real runs of the old build are kept as runs/*.instance. Each one is replayed against the new build, with the client library, SDK or Go bumped. Runs 1 and 2 matched. Run 3 diverged with stdout line 4 recorded as “total 12.50” and replayed as “total 12.5”. Run 4 diverged because a POST request isn’t in the recording.

Replay Every Run Against the New Build

Make the upgrade and build the new version at the same path, ./app. Replay runs the command stored in each file, so every run now goes through the new build. With -json the report is easy to handle in a script:

for f in runs/*.instance; do
  codeinstances replay -json "$f" > "${f%.instance}.json" || echo "diverged: $f"
done

If the new build lives somewhere else, give its command after -- for each file instead. replay exits 0 when a run matched and 1 when it diverged, so the loop can flag runs without reading the JSON. The JSON has matched, the list of divergences, and how many recorded calls were served.

A run that matched did the same thing: the same requests, the same output, the same exit code. A run that diverged says how, in plain words. “stdout line 4: recorded …, replayed …” means the program printed something different, which after a library bump is often number formatting or field order. “request POST … isn’t in the recording” means the new build sent a request the old one didn’t, maybe a new endpoint version or a changed query parameter. “recorded call 2 … was never made” means it skipped one. “exit code: recorded 0, replayed 1” means it now fails.

Each of those points at a specific behaviour change, which is a lot easier to judge than a changelog. Some are fine, and some are bugs. Headers aren’t compared at all, since replay matches on method, URL and body, so a new header the SDK adds won’t show up.

When It Doesn’t Fit

Replay only helps while the new build sends the same requests. If the upgrade changes every URL (an SDK that moves from /v1 to /v2, say), every run diverges on its first request and the report tells you little more than that. Then you record again with the new build and compare the outputs by hand. Request order doesn’t matter by default, since threads race, so a new client that reorders concurrent calls still matches. Pass -strict if you want order checked too. If stderr carries timings or warnings that change, -ignore-stderr leaves it out. And for Go upgrades, remember that the clock and random numbers only replay exactly if the program uses the hook package.