01Sources
50 bug-fix PRs from 5 open-source repositories
Real merged fixes from Cal.com (TypeScript), Sentry (Python), Grafana (Go), Keycloak (Java) and Discourse (Ruby). All five major server-side languages are represented.
AI Code Reviewer Benchmarks
How well Luxor Agent Cloud does versus other AI code reviewers on 122 real production bugs across 50 pull requests, graded per finding by a 3‑judge LLM panel.
In this independent 2026 benchmark of Luxor Agent Cloud and 6 other leading tools, Luxor Agent Cloud catches 84 of 122 real, production bugs across 50 pull requests from Cal.com, Sentry, Grafana, Keycloak and Discourse, graded per finding by a 3‑judge LLM panel.
84/ 122
Real bugs caughtEvery pull request contains a real, merged bug-fix from an open-source codebase. No synthetic bugs, no repo-specific tuning.
01Sources
Real merged fixes from Cal.com (TypeScript), Sentry (Python), Grafana (Go), Keycloak (Java) and Discourse (Ruby). All five major server-side languages are represented.
02Replay
The pre-fix diff is replayed against every tool with default settings: no custom rules, no repo-specific tuning, full repository context. Every tool sees the same code at the same point in history.
03Scoring
A bug counts as caught only if a line-level comment pinpoints the faulty code and explains its impact, and at least two of three independent LLM judges agree.
Test sources
Each bug is scored individually, not each pull request. Sorted by recall, every tool at its default configuration. More comments from Luxor Agent Cloud drive both its higher recall and its lower precision.
RecallShare of 122 real-world bugs caught
Scale 0–80% | Precision · F1 below each reviewer
Critical bugs cause outages, data loss or auth bypass. High‑severity bugs break major user-facing flows. Medium bugs degrade behavior without breaking it.
Catch rate by the severity of each individual finding.
| Reviewer | All122 bugs | Critical7 bugs | High53 bugs | Medium52 bugs | Low10 bugs |
|---|---|---|---|---|---|
| Luxor Agent Cloud | 69% | 57% | 74% | 69% | 50% |
| Devin | 36% | 29% | 45% | 29% | 30% |
| Greptile | 36% | 71% | 32% | 37% | 30% |
| Cursor | 32% | 14% | 34% | 35% | 20% |
| CodeRabbit | 29% | 43% | 25% | 33% | 20% |
| Copilot | 25% | 43% | 25% | 23% | 20% |
| Graphite | 3% | 0% | 2% | 4% | 10% |
Share of real bugs each reviewer caught in each codebase.
| Reviewer | Cal.com33 bugs | Discourse22 bugs | Grafana21 bugs | Keycloak15 bugs | Sentry31 bugs |
|---|---|---|---|---|---|
| Luxor Agent Cloud | 91% | 73% | 52% | 67% | 55% |
| Devin | 33% | 27% | 67% | 27% | 29% |
| Greptile | 30% | 50% | 52% | 53% | 13% |
| Cursor | 30% | 32% | 52% | 47% | 13% |
| CodeRabbit | 21% | 36% | 43% | 40% | 16% |
| Copilot | 21% | 32% | 33% | 47% | 6% |
| Graphite | 0% | 5% | 14% | 0% | 0% |
One row per real bug, 122 findings across 50 production pull requests. Each cell shows whether that reviewer flagged that specific defect, decided by 3‑LLM majority vote.
Get Luxor Agent Cloud
The cloud for your code and agents.
Set up Sandbox: luxor.ai/agents/sandbox.md