Skip to content

Accuracy benchmark ​

How often does fixlock get it wrong? We locked real bug fixes from real projects, edited each lock in ways that do and don't change what the code does, and checked every verdict.

For real bug fixes undone in full, as an AI port from older code would do, see the Regression Bench.

Results ​

Text matching (≤ 0.5)Structural matching (0.6+)
False positives: harmless edits flagged as a change55.7% (1622 of 2914)0.0% (0 of 2914)
Detection: harmful edits caught92.3% (1704 of 1846)99.9% (1845 of 1846)
Variable renames blocked as a change99.8%2.9% (96.9% reported as a rename)

764 locks · 5271 edits · 11 repositories · fixlock 0.6.0+bdadb1b · 2026-11-14

Corpus ​

Locks are the past fixes fixlock discover suggests for each repository (score 4+, the default), up to 80 per repo.

RepositoryLanguageCommitFixes discoveredLocks benchmarked
expressJavaScript43fca524635
kyTypeScript354188816880
requestsPython3c7baaa6131
cobraGoadbc88113580
gsonJavad938a1016780
okhttpKotlinb6e74e514380
ripgrepRust178cc027458
rackRubyda9046412880
guzzlePHPf8aca4027780
Newtonsoft.JsonC#52fa3ae29180
AlamofireSwiftdea4bab17180

Harmless edits (should not be flagged) ​

EditLanguagesEditsText flagsStructural flags
add commentsC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript76489.8%0.0%
reindentC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript7290.0%0.0%
wrap long lineC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript52091.5%0.0%
blank linesC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript3000.0%0.0%
drop semicolonsC#, Java, JavaScript, PHP, Rust, TypeScript26491.3%0.0%
prettier (narrow, single quotes, no semicolons)JavaScript, TypeScript11192.8%0.0%
prettier (wide, double quotes)JavaScript, TypeScript11154.1%0.0%
swap quote styleJavaScript, TypeScript41100.0%0.0%
black (line length 40)Python2853.6%0.0%
black (line length 120)Python260.0%0.0%
gofmt (after re-indenting with spaces)Go200.0%0.0%

Harmful edits (must be caught) ​

EditLanguagesEditsText catchesStructural catches
delete a lineC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript64991.7%99.8%
change fieldC#, Go, Java, JavaScript, Kotlin, Python, Ruby, Rust, Swift, TypeScript30292.4%100.0%
change stringC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript24593.9%100.0%
flip comparisonC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript24491.4%100.0%
swap and/orC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript15894.9%100.0%
drop negationC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript14491.7%100.0%
change numberC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript10491.3%100.0%

Renames ​

Renaming a local variable everywhere in the file keeps the logic but can hide a swap (isAdmin → isGuest). Structural matching reports it as a rename: a warning by default, configurable with matching.renames. Text matching blocks it as a change.

EditLanguagesEditsText: blockedStructural: blockedStructural: reported as rename
rename a local variableC#, Go, Java, JavaScript, Kotlin, PHP, Python, Ruby, Rust, Swift, TypeScript51199.8%2.9%96.9%

Where structural matching gets it wrong ​

  • Warned as a rename, not blocked: delete a line in gson metrics/src/main/java/com/google/gson/metrics/ParseBenchmark.java:367-367
  • Rename blocked as a change: rename a local variable in ky source/utils/timeout.ts:43-44
  • Rename blocked as a change: rename a local variable in ky source/core/Ky.ts:539-539
  • Rename blocked as a change: rename a local variable in ky source/core/Ky.ts:895-895
  • Rename blocked as a change: rename a local variable in requests pyproject.toml:58-58
  • Rename blocked as a change: rename a local variable in requests src/requests/_types.py:89-89
  • Rename blocked as a change: rename a local variable in requests src/requests/_types.py:158-158
  • Rename blocked as a change: rename a local variable in cobra command.go:1488-1488
  • Rename blocked as a change: rename a local variable in gson gson/src/main/java/com/google/gson/internal/GsonTypes.java:405-405

A harmful edit "warned as a rename" means the remaining code had the same shape as the lock with different names, so fixlock reported a rename instead of a change: the PR still shows a warning. A rename is blocked as a change when the lock is very short (under 4 code tokens) or the old name is still used nearby; that's deliberate, because those are the cases where a "rename" is often a different variable.

Full details for every case are in bench/results.json.

Method ​

  1. Each repository is fetched at the pinned commit and fixlock discover picks its past fixes. A lock is used only if both matchers agree it is intact before any edit.
  2. Each lock gets every edit that applies to it. Edits only touch the lock's own lines, except renames and formatters, which change the whole file.
    • Harmless: add comments, add blank lines, wrap a long line inside brackets, swap ' / " (JS/TS), drop semicolons (JS/TS), re-indent (not Python); Prettier with two opposite configs (JS/TS), Black at line length 40 and 120 (Python), gofmt (Go).
    • Harmful: flip a comparison, swap &&/|| (and/or), drop a negation, change a number, change a string, read a different field, delete the main line of the fix.
  3. Both matchers judge the edited file. A harmless edit judged changed is a false positive. A harmful edit judged anything but changed is a miss.

Reproduce: node bench/run.js (needs git, network for the first fetch, and optionally prettier, black and gofmt on PATH). Runs in about 230 seconds.

Limits ​

These are mechanical edits, not a sample of real pull requests; real changes mix several edits at once. Formatter results depend on the formatter versions. A consistent rename that changes meaning (isAdmin → isGuest) is reported as a rename, not blocked, unless matching.renames is set to block.

MIT licensed · A product of Algotricz