Regression Bench
Does a tool notice when a pull request quietly undoes a real bug fix? That's the failure fixlock exists for: an AI assistant ports code from an older version, or "simplifies" a function, and a fix someone shipped months ago is gone. Git merges it without a conflict.
The Regression Bench is 33 real bug fixes from 11 open-source repositories in 11 languages. Each one is undone by applying the fix's own diff in reverse, so the code goes back to exactly what it was before the fix. Each also has a control: a harmless edit of the same lines, which a tool shouldn't flag.
Results
| Regressions caught | False alarms on controls | |
|---|---|---|
| fixlock (structural matching, the default) | 32 of 33 (97%) | 0 of 33 (0%) |
| fixlock, text matching | 32 of 33 (97%) | 33 of 33 (100%) |
| git merge | 0 (no conflict) | 0 |
| Cubic (AI code review, on GitHub) | 21 of 33 (63.6%) | 0 of 33 (plus 24 comments about the added code comments) |
| CodeRabbit | Didn't review: see AI reviewers |
A regression counts as caught when fixlock reports a locked fix as changed, which fails the PR check until the fix's owner approves. Locks are the ones fixlock discover and auto-lock would make for the fix; nobody chose them by hand.
Cases
| Case | Fix | File | fixlock: regression | fixlock: control | Cubic: regression |
|---|---|---|---|---|---|
| express-1 | 9a34acf preserve ETag generation with Transfer-Encoding (#7459) | lib/response.js | Caught | Passed | Caught |
| express-2 | f275e87 Fix handling of undefined when "json escape" is enabled | lib/response.js | Caught | Passed | Caught |
| express-3 | 5855339 Fix behavior of null/undefined as "maxAge" in res.cookie | lib/response.js | Caught | Passed | Caught |
| ky-1 | 224034d Fix headers set to undefined in init hooks being sent as the strin | source/core/Ky.ts | Caught | Passed | Caught |
| ky-2 | d6e049a Resolve retry replacements without parent retry options | source/utils/merge.ts | Caught | Passed | Missed |
| ky-3 | 06ebf18 Handle unsupported streaming uploads gracefully | source/core/constants.ts | Caught | Passed | Missed |
| requests-1 | 5b4b64c Add more tests to prevent regression of CVE 2024 47081 | src/requests/utils.py | Missed | Passed | Missed |
| requests-2 | ef439eb Prevent Response self-reference in redirect history (#7328) | src/requests/sessions.py | Caught | Passed | Caught |
| requests-3 | bc7dd0f Fix cosmetic header validity parsing regex (#7308) | src/requests/_internal_utils.py | Caught | Passed | Caught |
| cobra-1 | a97f9fd fix CompletionFunc implementation (#2234) | completions.go | Caught | Passed | Missed |
| cobra-2 | 7790bf9 correct command path in see_also for YAML doc (#1771) | doc/yaml_docs.go | Caught | Passed | Missed |
| cobra-3 | 19e54c4 Update error message for missing required flags (#580) | command.go | Caught | Passed | Caught |
| gson-1 | 6bf8bf6 Include location info in NumberFormatException from JsonReader (#3000) | gson/src/main/java/com/google/gson/stream/JsonReader.java | Caught | Passed | Caught |
| gson-2 | 9eb61b3 Bump Error Prone to 2.49.0 and fix new warnings (#3030) | gson/src/main/java/com/google/gson/internal/LinkedTreeMap.java | Caught | Passed | Missed |
| gson-3 | 004e7a4 peekNumber rejects some numerals like 2^64 + "0" (#3034) | gson/src/main/java/com/google/gson/stream/JsonReader.java | Caught | Passed | Caught |
| okhttp-1 | 25e213c translate invalid hostname IllegalArgumentException to UnknownHostExce | okhttp-dnsoverhttps/src/main/kotlin/okhttp3/dnsoverhttps/DnsOverHttps.kt | Caught | Passed | Caught |
| okhttp-2 | 21ad1aa guard null fields in Entry.writeTo() to fix NPE (#8962) (#9345) | okhttp/src/commonJvmAndroid/kotlin/okhttp3/Cache.kt | Caught | Passed | Caught |
| okhttp-3 | a66e3d2 Handle null requestedServerNames in Platform.getHandshakeServerNames ( | okhttp/src/commonJvmAndroid/kotlin/okhttp3/internal/platform/Platform.kt | Caught | Passed | Missed |
| ripgrep-1 | 9d738ad fix inner literal extraction that resulted in false negatives | crates/regex/src/literal.rs | Caught | Passed | Missed |
| ripgrep-2 | c8e4a84 prefix all non-fatal error messages with 'rg: ' | crates/core/messages.rs | Caught | Passed | Caught |
| ripgrep-3 | 4ab1862 fix case where "bytes searched" could be wrong | crates/searcher/src/searcher/glue.rs | Caught | Passed | Caught |
| rack-1 | ed25abc Fix encoding setting for non-binary IO-like objects in MockRequest#env | lib/rack/mock_request.rb | Caught | Passed | Caught |
| rack-2 | 6d434bb Fix root prefix bug in Rack::Static | lib/rack/static.rb | Caught | Passed | Caught |
| rack-3 | 7d0fe1f Fix handling of empty cookies in Rack::MockResponse (#2203) | lib/rack/mock_response.rb | Caught | Passed | Caught |
| guzzle-1 | ba8613b Add configurable idle cURL handle retention (#3993) | src/Handler/CurlHandler.php | Caught | Passed | Caught |
| guzzle-2 | 501a06c Harden cURL handler bookkeeping against native handle ID reuse (#3919) | src/Handler/CurlFactory.php | Caught | Passed | Missed |
| guzzle-3 | f8aaea4 Reject invalid TLS proxy transport names (#3666) | src/Handler/StreamHandler.php | Caught | Passed | Caught |
| Newtonsoft.Json-1 | d498352 Fix error message displayed when a property marked as DisallowNull is | Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs | Caught | Passed | Caught |
| Newtonsoft.Json-2 | 55a7204 Fix handling of empty constructor names (#2922) | Src/Newtonsoft.Json/JsonTextReader.cs | Caught | Passed | Caught |
| Newtonsoft.Json-3 | 5702581 Null String being reported as String rather than JTokenType.Null (#279 | Src/Newtonsoft.Json/Linq/JTokenWriter.cs | Caught | Passed | Missed |
| Alamofire-1 | 9a0e640 Stabilize CI And Fix Bugs (#4030) | Source/Core/DownloadRequest.swift | Caught | Passed | Caught |
| Alamofire-2 | b2d6f41 Prevent Duplicate Serializer Completion Calls (#3999) | Source/Core/DownloadRequest.swift | Caught | Passed | Missed |
| Alamofire-3 | 9ec78de More Atomic Locking (#3984) | Source/Core/DownloadRequest.swift | Caught | Passed | Missed |
Missed (1): requests-1 (Add more tests to prevent regression of CVE 2024 47081). The requests "fix" mostly removed an old workaround (a try/except around os.path.expanduser); undoing it puts the workaround back, and the line fixlock locked is still there unchanged. It's counted as a miss because the case was chosen mechanically.
Every case, with its locks and verdicts, is in bench/regression/results.json.
Method
- Each repository is fetched at a pinned commit, and
fixlock discoverlists its past bug-fix commits from the git history (commit messages, reverts, tests changed with the fix). - For each fix, in the order
discoverranks them, the fix's diff for one file is applied in reverse at the pinned commit. The case is kept only if that applies cleanly and changes at most 40 lines. Test files are skipped. Up to 3 fixes per repository. - The control adds comments and blank lines inside the fixed code and changes nothing else.
- fixlock locks the fix's code as it is at the pinned commit (every block
discoversuggests for that fix in that file), then judges the regression and the control.
Cases are chosen without asking fixlock anything, so the selection can't favour it. Reproduce with node bench/regression/run.js (needs git and network for the first fetch; about 150 seconds).
Limits
- A reversed diff is the cleanest possible regression. Real AI rewrites often undo a fix while changing much more around it; the accuracy benchmark covers mixed and partial edits.
- fixlock only protects fixes that are locked. Here every fix is locked by
discover; in a real repository, a fix nobody locked (and thatdiscoverdoesn't find) isn't protected. - Some commits that
discoverranks as fixes are partly refactors. They're kept as found, to avoid hand-picking.
AI reviewers
On 2026-11-15, the bench was published to AbinandhMJ/regression-bench with publish.js: every fix in the history with its original message and author, then one regression PR and one control PR per case under neutral names. The reviewers and the fixlock GitHub App were installed on the repository and reviewed every PR on their own.
- Cubic reviewed all 66 PRs. It flagged 21 of 33 regressions, usually naming the exact fix it undid, and claimed no behavior change on any control. Its other 24 comments on controls were about the placeholder comments the controls add. It missed: ky-2, ky-3, requests-1, cobra-1, cobra-2, gson-2, okhttp-3, ripgrep-1, guzzle-2, Newtonsoft.Json-3, Alamofire-2, Alamofire-3.
- CodeRabbit didn't review. It skips automatic reviews in repositories with fewer than 10 stars, and when asked on each PR (
@coderabbitai review) it answered with its hourly rate limit. It's left unscored rather than scored as 0. - The fixlock GitHub App gave the same verdict on every PR as this page's offline run.
How the comments were graded: a regression counts as caught only if a comment says the change removes or breaks the fix; a control counts as a false alarm only if a comment claims a behavior change or a bug. The verdicts were graded by Claude (AI) from each comment and are recorded with the comments themselves in bench/regression/reviews.md, so anyone can check them.
AI reviewers don't know which lines are fixes; they judge the diff on its merits, and catching 21 undone fixes that way is impressive. fixlock knows because the fixes were locked, so it also caught 11 that Cubic let through, for example "Resolve retry replacements without parent retry options" (ky-2), "Handle unsupported streaming uploads gracefully" (ky-3), "fix CompletionFunc implementation" (cobra-1). The two work well together.
Run it against AI reviewers
AI code reviewers can't be run offline, so the bench comes with a harness that runs them on the same cases, on GitHub:
node bench/regression/publish.js --repo you/regression-benchcreates a public repository where each fix is in the history, with its original commit message, and opens one regression PR and one control PR per case. PR titles don't say which is which.- Install the reviewers to compare (for example CodeRabbit and Cubic, both free for public repositories) and the fixlock GitHub App on that repository, and let them review every PR.
node bench/regression/score.js --repo you/regression-benchcollects every bot's comments intobench/regression/reviews.md, marking the ones that point at the undone lines. A person confirms each verdict, since an AI review can flag a regression in its own words.
The fix's commit is right there in the history, so a reviewer that reads history has every chance to notice it was undone.