Skip to content

Regression Bench ​

Does a tool notice when a pull request quietly undoes a real bug fix? That's the failure fixlock exists for: an AI assistant ports code from an older version, or "simplifies" a function, and a fix someone shipped months ago is gone. Git merges it without a conflict.

The Regression Bench is 33 real bug fixes from 11 open-source repositories in 11 languages. Each one is undone by applying the fix's own diff in reverse, so the code goes back to exactly what it was before the fix. Each also has a control: a harmless edit of the same lines, which a tool shouldn't flag.

Results ​

Regressions caughtFalse alarms on controls
fixlock (structural matching, the default)32 of 33 (97%)0 of 33 (0%)
fixlock, text matching32 of 33 (97%)33 of 33 (100%)
git merge0 (no conflict)0
Cubic (AI code review, on GitHub)21 of 33 (63.6%)0 of 33 (plus 24 comments about the added code comments)
CodeRabbitDidn't review: see AI reviewers

A regression counts as caught when fixlock reports a locked fix as changed, which fails the PR check until the fix's owner approves. Locks are the ones fixlock discover and auto-lock would make for the fix; nobody chose them by hand.

Cases ​

CaseFixFilefixlock: regressionfixlock: controlCubic: regression
express-19a34acf preserve ETag generation with Transfer-Encoding (#7459)lib/response.jsCaughtPassedCaught
express-2f275e87 Fix handling of undefined when "json escape" is enabledlib/response.jsCaughtPassedCaught
express-35855339 Fix behavior of null/undefined as "maxAge" in res.cookielib/response.jsCaughtPassedCaught
ky-1224034d Fix headers set to undefined in init hooks being sent as the strinsource/core/Ky.tsCaughtPassedCaught
ky-2d6e049a Resolve retry replacements without parent retry optionssource/utils/merge.tsCaughtPassedMissed
ky-306ebf18 Handle unsupported streaming uploads gracefullysource/core/constants.tsCaughtPassedMissed
requests-15b4b64c Add more tests to prevent regression of CVE 2024 47081src/requests/utils.pyMissedPassedMissed
requests-2ef439eb Prevent Response self-reference in redirect history (#7328)src/requests/sessions.pyCaughtPassedCaught
requests-3bc7dd0f Fix cosmetic header validity parsing regex (#7308)src/requests/_internal_utils.pyCaughtPassedCaught
cobra-1a97f9fd fix CompletionFunc implementation (#2234)completions.goCaughtPassedMissed
cobra-27790bf9 correct command path in see_also for YAML doc (#1771)doc/yaml_docs.goCaughtPassedMissed
cobra-319e54c4 Update error message for missing required flags (#580)command.goCaughtPassedCaught
gson-16bf8bf6 Include location info in NumberFormatException from JsonReader (#3000)gson/src/main/java/com/google/gson/stream/JsonReader.javaCaughtPassedCaught
gson-29eb61b3 Bump Error Prone to 2.49.0 and fix new warnings (#3030)gson/src/main/java/com/google/gson/internal/LinkedTreeMap.javaCaughtPassedMissed
gson-3004e7a4 peekNumber rejects some numerals like 2^64 + "0" (#3034)gson/src/main/java/com/google/gson/stream/JsonReader.javaCaughtPassedCaught
okhttp-125e213c translate invalid hostname IllegalArgumentException to UnknownHostExceokhttp-dnsoverhttps/src/main/kotlin/okhttp3/dnsoverhttps/DnsOverHttps.ktCaughtPassedCaught
okhttp-221ad1aa guard null fields in Entry.writeTo() to fix NPE (#8962) (#9345)okhttp/src/commonJvmAndroid/kotlin/okhttp3/Cache.ktCaughtPassedCaught
okhttp-3a66e3d2 Handle null requestedServerNames in Platform.getHandshakeServerNames (okhttp/src/commonJvmAndroid/kotlin/okhttp3/internal/platform/Platform.ktCaughtPassedMissed
ripgrep-19d738ad fix inner literal extraction that resulted in false negativescrates/regex/src/literal.rsCaughtPassedMissed
ripgrep-2c8e4a84 prefix all non-fatal error messages with 'rg: 'crates/core/messages.rsCaughtPassedCaught
ripgrep-34ab1862 fix case where "bytes searched" could be wrongcrates/searcher/src/searcher/glue.rsCaughtPassedCaught
rack-1ed25abc Fix encoding setting for non-binary IO-like objects in MockRequest#envlib/rack/mock_request.rbCaughtPassedCaught
rack-26d434bb Fix root prefix bug in Rack::Staticlib/rack/static.rbCaughtPassedCaught
rack-37d0fe1f Fix handling of empty cookies in Rack::MockResponse (#2203)lib/rack/mock_response.rbCaughtPassedCaught
guzzle-1ba8613b Add configurable idle cURL handle retention (#3993)src/Handler/CurlHandler.phpCaughtPassedCaught
guzzle-2501a06c Harden cURL handler bookkeeping against native handle ID reuse (#3919)src/Handler/CurlFactory.phpCaughtPassedMissed
guzzle-3f8aaea4 Reject invalid TLS proxy transport names (#3666)src/Handler/StreamHandler.phpCaughtPassedCaught
Newtonsoft.Json-1d498352 Fix error message displayed when a property marked as DisallowNull isSrc/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.csCaughtPassedCaught
Newtonsoft.Json-255a7204 Fix handling of empty constructor names (#2922)Src/Newtonsoft.Json/JsonTextReader.csCaughtPassedCaught
Newtonsoft.Json-35702581 Null String being reported as String rather than JTokenType.Null (#279Src/Newtonsoft.Json/Linq/JTokenWriter.csCaughtPassedMissed
Alamofire-19a0e640 Stabilize CI And Fix Bugs (#4030)Source/Core/DownloadRequest.swiftCaughtPassedCaught
Alamofire-2b2d6f41 Prevent Duplicate Serializer Completion Calls (#3999)Source/Core/DownloadRequest.swiftCaughtPassedMissed
Alamofire-39ec78de More Atomic Locking (#3984)Source/Core/DownloadRequest.swiftCaughtPassedMissed

Missed (1): requests-1 (Add more tests to prevent regression of CVE 2024 47081). The requests "fix" mostly removed an old workaround (a try/except around os.path.expanduser); undoing it puts the workaround back, and the line fixlock locked is still there unchanged. It's counted as a miss because the case was chosen mechanically.

Every case, with its locks and verdicts, is in bench/regression/results.json.

Method ​

  1. Each repository is fetched at a pinned commit, and fixlock discover lists its past bug-fix commits from the git history (commit messages, reverts, tests changed with the fix).
  2. For each fix, in the order discover ranks them, the fix's diff for one file is applied in reverse at the pinned commit. The case is kept only if that applies cleanly and changes at most 40 lines. Test files are skipped. Up to 3 fixes per repository.
  3. The control adds comments and blank lines inside the fixed code and changes nothing else.
  4. fixlock locks the fix's code as it is at the pinned commit (every block discover suggests for that fix in that file), then judges the regression and the control.

Cases are chosen without asking fixlock anything, so the selection can't favour it. Reproduce with node bench/regression/run.js (needs git and network for the first fetch; about 150 seconds).

Limits ​

  • A reversed diff is the cleanest possible regression. Real AI rewrites often undo a fix while changing much more around it; the accuracy benchmark covers mixed and partial edits.
  • fixlock only protects fixes that are locked. Here every fix is locked by discover; in a real repository, a fix nobody locked (and that discover doesn't find) isn't protected.
  • Some commits that discover ranks as fixes are partly refactors. They're kept as found, to avoid hand-picking.

AI reviewers ​

On 2026-11-15, the bench was published to AbinandhMJ/regression-bench with publish.js: every fix in the history with its original message and author, then one regression PR and one control PR per case under neutral names. The reviewers and the fixlock GitHub App were installed on the repository and reviewed every PR on their own.

  • Cubic reviewed all 66 PRs. It flagged 21 of 33 regressions, usually naming the exact fix it undid, and claimed no behavior change on any control. Its other 24 comments on controls were about the placeholder comments the controls add. It missed: ky-2, ky-3, requests-1, cobra-1, cobra-2, gson-2, okhttp-3, ripgrep-1, guzzle-2, Newtonsoft.Json-3, Alamofire-2, Alamofire-3.
  • CodeRabbit didn't review. It skips automatic reviews in repositories with fewer than 10 stars, and when asked on each PR (@coderabbitai review) it answered with its hourly rate limit. It's left unscored rather than scored as 0.
  • The fixlock GitHub App gave the same verdict on every PR as this page's offline run.

How the comments were graded: a regression counts as caught only if a comment says the change removes or breaks the fix; a control counts as a false alarm only if a comment claims a behavior change or a bug. The verdicts were graded by Claude (AI) from each comment and are recorded with the comments themselves in bench/regression/reviews.md, so anyone can check them.

AI reviewers don't know which lines are fixes; they judge the diff on its merits, and catching 21 undone fixes that way is impressive. fixlock knows because the fixes were locked, so it also caught 11 that Cubic let through, for example "Resolve retry replacements without parent retry options" (ky-2), "Handle unsupported streaming uploads gracefully" (ky-3), "fix CompletionFunc implementation" (cobra-1). The two work well together.

Run it against AI reviewers ​

AI code reviewers can't be run offline, so the bench comes with a harness that runs them on the same cases, on GitHub:

  1. node bench/regression/publish.js --repo you/regression-bench creates a public repository where each fix is in the history, with its original commit message, and opens one regression PR and one control PR per case. PR titles don't say which is which.
  2. Install the reviewers to compare (for example CodeRabbit and Cubic, both free for public repositories) and the fixlock GitHub App on that repository, and let them review every PR.
  3. node bench/regression/score.js --repo you/regression-bench collects every bot's comments into bench/regression/reviews.md, marking the ones that point at the undone lines. A person confirms each verdict, since an AI review can flag a regression in its own words.

The fix's commit is right there in the history, so a reviewer that reads history has every chance to notice it was undone.

MIT licensed · A product of Algotricz