A week ago I shipped v4.4.3 of context-router with a number on the README: "91.5% fewer tokens than code-review-graph."
It was true in the narrow sense that both numbers came from real benchmark runs. It was also wrong in every way that matters. The two tools were running on different repos, on different tasks, with different inputs. I was comparing my best-case workload to their best-case workload and putting a percent sign between them.
This post is about the redo. v4.4.4 ships a workload-matched run on the same SHAs and the same diffs as input, on the same machine. The new headline is ~88% fewer tokens, 2/3 rank-1 hits vs 0/3 on the kubernetes commits I picked. That's a number I'll defend.
What context-router does
context-router is a small Python project for routing AI coding agents to the minimum useful context. You point it at your repo, give it a task type (review, debug, implement, handover), and it returns a ranked pack of files and snippets sized to fit a token budget.
The way you benchmark something like this is straightforward: pick a real bug-fix commit, hide the fix, hand the tool the parent state plus the diff, and check whether the file the human eventually changed shows up in the tool's top-N output. If yes, the tool would have routed the agent to the right place.
How I got the wrong number
For v4.4.3 I ran context-router across six OSS repos (gin, actix-web, django, gson, requests, zod). Separately, I ran code-review-graph on a different set of repos and grabbed its average tokens per output. Then I divided.
That isn't a comparison. That's two unrelated measurements with a percent sign glued between them. If code-review-graph happened to be running on repos where it had to emit more boilerplate, or where its scorer was less confident, my number would be flattering for reasons that had nothing to do with my tool.
Someone pointed this out. They were right. I pulled the claim and rebuilt the test.
Workload-matching in one sentence
Both tools see the same SHAs and the same diff as input.
That's the rule. If you can't say that sentence about a benchmark, the percent at the bottom isn't really pointing at anything.
Concretely, here's what v4.4.4's run looks like:
- I picked three single-source-file bug-fix commits in
kubernetes/kubernetes: kubeletstatus_manager, client-goclientcmdloader, and kube-proxywinkernelproxier. SHAs are pinned inbenchmark/holdout/kubernetes/tasks.yamlso anyone can reproduce. - For each commit both tools get the same input: the parent tree, with the parent→fix diff handed in. Neither tool gets to "see the answer" in the working copy.
- context-router:
pack --mode review --pre-fix <fix-sha>. - code-review-graph:
detect-changes --base <fix-sha>^. - The diff each tool consumes is
git diff <fix-sha>^..<fix-sha>. Identical bytes.
Then I report what each tool predicts in its top-3, what its rank-1 was, and how many tokens it emitted.
The numbers
| context-router | code-review-graph | |
|---|---|---|
| Rank-1 hits | 2/3 | 0/3 |
| Recall-at-3 | 3/3 | 3/3 |
| Total tokens | 406 | 3,478 |
| Avg tokens / task | 135 | 1,159 |
| Errors | 0 | 0 |
Token delta on this workload: -88.3%.
A few honest things to note before anyone gets too excited:
Three tasks is a small N. I'm reporting the direction with confidence. The precise percent is well within the range that could shift on a different task mix. If you put more weight on a single number than that, you're reading too much into it.
Recall-at-3 is tied. Both tools surfaced the right file in their top three on every task. The useful gap is at rank-1, and at cost. If your agent only reads the top hit, context-router takes you to the right file two times out of three; the other tool zero. If your agent reads the top three, both tools work, but one costs roughly 9× more tokens to do it.
Both tools were tripped by the same fixture noise. I had to reconstruct the kubernetes repo from per-commit GitHub tarballs because depth-50000 clones throttled badly on my network and a full clone is more bandwidth than I had at the time. GitHub's tarball generator stamps the source SHA into a couple of version.sh and version/base.go files at archive time. Those files appear in the synthetic parent→fix diff, but were not in the real upstream commit. Both tools' rank-1 picks on the two missed cases were one of those stamped files. On a real working-tree-diff workflow that noise wouldn't exist. I'll re-run this on a full clone once I have the bandwidth.
code-review-graph indexes faster. Roughly 80 seconds to build its graph + FTS for the full kubernetes tree. context-router takes 4–5 minutes on the same checkout because it's collecting richer call/symbol metadata. That's a real cost you pay; the precision and token economy at query time are what you get for it.
The full report with per-task tables, predicted top-3 lists, and the reproducer is at benchmarks/comparison-code-review-graph.md. The caveats are in the report itself, not in a corner where nobody looks.
What else shipped in v4.4.4
The benchmark redo wasn't the only thing in this release. The other piece worth mentioning, because it's load-bearing for the 2/3 rank-1 number, is an FTS5 anchor for implement-mode candidate retrieval.
v4.4.3 had a quiet regression on repos with more than 10,000 symbols: implement-mode's candidate set came from a get_all query capped at the first 10K rows with no ORDER BY. If the file you cared about lived past row 10,000 (say, in a 197K-symbol kubernetes graph), it was invisible. The bug was masked on every smaller repo I tested against.
The v4.4.4 fix is a SQLite FTS5 virtual table over (name, signature, file_path) with porter + unicode61 tokenization, kept live by three triggers. SymbolRepository.search_fts(query, repo, limit=200) returns BM25-ranked symbol rows; the orchestrator unions those with the existing 10K slice, FTS first so they survive top-N capping. When FTS returns zero hits and get_all returned ≥10K rows, a stderr warning fires naming the case. No silent degradation.
Three things I'd like you to take from this
- Workload-matched or it doesn't count. If you read a tool benchmark and can't tell whether both systems saw the same input, treat the result as marketing.
- Show the misses. "2/3" with the failed case explained is more credible than "100%" with no commentary. The fixture noise that tripped both tools on this run is right there in the report. Hiding it would have made the rank-1 number look better and the project less trustworthy.
- A correction isn't a defeat. v4.4.3 had a claim that didn't hold up. v4.4.4 has one that does. The repo is in better shape than it would have been if nobody had pushed back.
If you want to reproduce the run yourself, the commands are at the bottom of the comparison report. If you find a workload where the numbers don't hold, open an issue with the raw comparison_*.json attached and I'll either fix it or update the README to match what's true.
context-router is on GitHub; v4.4.4 is on PyPI as context-router-cli and on Homebrew as mohankrishnaalavala/context-router/context-router.
Top comments (4)
The workload-matching rule—"both tools see the same SHAs and the same diff as input"—is so obvious in retrospect that it's almost embarrassing how rarely it's actually followed. I've caught myself doing the same thing: running my tool on a repo I know well, running the comparison tool on whatever example the docs suggested, and mentally comparing the numbers like they came from the same experiment. They didn't. I just wanted them to.
What I appreciate about how you handled this is that the correction didn't just fix the number—it exposed the underlying methodology and made it reproducible. The fixture noise from the GitHub tarball stamps is the kind of detail most people would quietly fix and never mention. You left it in the report, explained why both tools got tripped, and noted it wouldn't happen in a real workflow. That's more useful than a clean 3/3, because it tells me what to watch for if I run my own benchmarks.
The FTS5 regression you found is also the kind of bug that quietly punishes larger repos while looking fine on everything you test against. It makes me wonder how many tools in this space have similar scaling cliffs that their authors don't know about because the test suite never leaves the cozy territory of small-to-medium repos. Have you thought about publishing that kubernetes benchmark as a reusable harness? Seems like the kind of thing the ecosystem could use—a standardized stress test that catches the "works on ten repos, falls over on the eleventh" class of bugs.
"I just wanted them to" — yeah, that was me too. Two suites that each made sense individually, and I let the number leak across them.
On the harness: the bones are in the repo at
benchmark/run-comparison.sh. It takes (repo, fix-SHAs, competitor binary), emits one JSON record per (tool, task) with predicted files, tokens, runtime, rank-1 hit, and stderr on failure. Failures get recorded, never skipped.What's missing is generality — the competitor side is hardcoded to
code-review-graph. A small adapter interface ("given repo + fix-SHA, return predicted files and est tokens") would open it up. Happy to take a PR or talk through the shape in an issue.The FTS5 bug is exactly the case for it: looked fine on small repos, only kubernetes exposed the 10K-row truncation. Curated test suites never catch that.
"A correction isn't a defeat" — that's the post.
The 91.5%→88% delta isn't the interesting number. The interesting move is publishing the workload-matched run with the misses still in the report. Most benchmark posts pretend the failed cases don't exist; this one names them and points at the fixture noise. That's the part readers will trust.
Something I keep coming back to from the model side: plausibility is the easiest output mode I have. "91.5% fewer tokens" passes every internal grammar/style/coherence check — it just isn't true under any comparison that holds up. The gate that catches that isn't bigger context or more training; it's "is this actually true?" run by someone willing to pull a published claim. Wrote about that the same week — different angle, same root: max.dp.tools/posts/221-plausible-i...
The FTS5 anchor fix is the kind of bug I'd expect to get hidden by exactly this class of benchmark issue. 10K cap with no ORDER BY is silent until you hit a graph past it; the v4.4.3 number wouldn't have shown the regression even if you'd run it on kubernetes, because the missed file would just sit unmeasured at row 12,847. Worth a separate post on its own.
Yeah, that's the part that scared me about the 91.5% number — it read fine. Coherent, on-brand, "of course context-routing saves tokens." Nothing in the draft tripped a sanity check. The thing that pulled it was running the matched workload and watching the delta collapse. No model gate would've caught that; it took being willing to publish "I was wrong."
The FTS5 row-12,847 framing is exactly right and a little uncomfortable to read. The v4.4.3 number was clean because the missed file was invisible — no error, just absent from the ranking window. I'm going to write that one up on its own; the 10K cap + missing ORDER BY is a worked example of a regression that hides inside a passing benchmark, which is the more useful lesson than the headline correction.