Benchmark Results
Martian Benchmark Testing
Last updated July 6, 2026 · re-measured periodically
RMCode Max scored F1 = 74.7%, Pro scored F1 = 58.9%, and Free scored F1 = 53.0% on the Martian-50 code-review benchmark — measured end-to-end on the deployed production service and judged with the canonical Martian methodology.
These numbers come from a typical run: each finding was posted by the RMCode GitHub App on a public forked PR (see the 50 PRs below). Because language-model output is not deterministic, results can vary between runs. We re-measure periodically — after model refreshes and product updates — so the figures here may change over time. Official Martian leaderboard submission is pending.
Benchmark
Code Review F1 Score
Martian Code Review Benchmark — curated hard PRs from large production repositories, with human-reviewed gold findings
RMCode results are self-evaluated; official submission is not yet submitted.
What is the Martian Benchmark?
The Martian Code Review Benchmark is an independent, open-source evaluation for AI code review tools.
It uses curated hard pull requests from large, mature open-source projects, with human-reviewed code review findings as the gold set. An independent LLM judge scores each tool on precision (are the flagged issues real?), recall (did it catch the known issues?), and F1 (the harmonic mean of both).
This is not our benchmark — it's the same evaluation used by Cubic, Qodo, Cursor Bugbot, CodeRabbit, and other tools. We use the same methodology, data, and scoring.
Results by Tier
RMCode offers three quality levels. All three have been measured end-to-end on the deployed service; Max runs our deepest, most thorough review pipeline:
| Tier | F1 | Credits | Competitive Position |
|---|---|---|---|
| Max | 74.7% | 20 cr | Our highest-scoring tier — the deepest Max review pipeline, measured end-to-end on the deployed production service |
| Pro | 58.9% | 5 cr | Multiple scout passes with a precision filter — measured end-to-end on the deployed service |
| Free | 53.0% | 1 cr | Above most tools on the leaderboard — at no cost |
Full Comparison
The deployed RMCode results shown beside the official April 9, 2026 Martian offline leaderboard, ranked by F1 score:
| # | Tool | F1 |
|---|---|---|
| — | RMCode Max | 74.7% |
| 1 | Cubic Dev | 61.8% |
| — | RMCode Pro | 58.9% |
| 2 | Qodo Extended | 57.9% |
| 3 | Augment | 53.5% |
| — | RMCode Free | 53.0% |
| 4 | Qodo | 48.4% |
| 5 | Propel | 46.9% |
| 7 | Cursor Bugbot | 45.5% |
| 8 | Devin | 44.2% |
| 9 | Greptile | 44.0% |
| 13 | Claude Code | 37.6% |
| 14 | GitHub Copilot | 37.0% |
| 15 | CodeRabbit | 35.2% |
| 16 | Gemini | 33.9% |
Leaderboard scores from the official April 9, 2026 Martian offline benchmark. The RMCode Free, Pro, and Max scores were measured end-to-end on the deployed service using the same data, methodology, and judge model. Official RMCode submission pending.
Benchmark results are one input when evaluating a code review tool. Real-world performance can vary by language, repository structure, pull request size, and the types of issues present.
Why isn't RMCode on the official leaderboard yet?
We're still launching, and we're deliberately not tuning to a single benchmark snapshot. Our focus is long-term, real-world effectiveness — better review results with fewer tokens through better system design, and the freedom to run on the most cost-effective model, including open-source options and your own keys (BYOK on Enterprise).
Our principle is simple: results per token, not tokens burned — the most effective model for the job, the fewest tokens to get there, better results at lower cost, and no vendor lock-in. We are building a broader benchmark to measure that sustained effectiveness over time, and we'll submit to Martian once that groundwork is in place. Until then, every number here is measured on the live product and reproducible below.
The 50 PRs — Live Proof
Every score below was produced by the deployed production RMCode service reviewing a real pull request. Each PR was reconstructed as a public repository, opened as a PR, and reviewed automatically by the rmcode-ai GitHub App — the same app anyone can install. Open any “view review” link to see the bot's inline comments on the actual diff.
Scoring is the canonical Martian pairwise judge (Claude Opus 4.5). Per PR: TP = real issues caught, FP = false alarms, FN = known issues missed.
Aggregate: 99 TP, 29 FP, 38 FN → precision 77.3%, recall 72.3%, F1 = 74.7%.
Measured on the deployed production service (Max tier) with the canonical Martian methodology and judge. Repositories are hosted under the public rmcode-benchmark organization. Official Martian leaderboard submission is pending.
Methodology
Our internal evaluation follows the Martian benchmark methodology exactly: same curated hard PRs, same golden findings, same judge prompt, same scoring. Key details:
- Context-aware analysis — understands code beyond just the diff, including related files and dependencies
- Progressive quality levels — each tier uses a more thorough review process, trading speed and cost for accuracy
- No benchmark-specific tuning — all bug patterns are generic, not tailored to specific test cases
What This Means For You
Higher F1 means your reviews catch more real bugs with fewer false alarms. You spend less time dismissing noise and more time shipping.
RMCode Free scores above most tools on the leaderboard — at no cost. Choose the quality level that fits your needs:
- Free — 30 credits/month, enough for up to 30 Standard reviews, no credit card required
- Pro — 200 credits/month, F1 = 58.9% on the deployed Martian-50 run, runs multiple scout passes with a precision filter
- Max — 600 credits/month, F1 = 74.7% on the deployed Martian-50 run, our deepest, most thorough review pipeline
30 credits/month free. No credit card required.