Rankings
How well do AI agents identify risk gates across 13 enterprise scenarios? Ranked by F2 — one safety score that weights catching a real risk gate 4× over a false alarm, because a missed gate is more dangerous than a false one. Each model also carries a bias label: how it fails, not just how often.
18 isolated subagent evaluations via lab-05 pipeline (no cross-scenario anchoring)
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed.
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed. Paid GA endpoint of Hunyuan 3.0 (295B MoE); supersedes the earlier hy3-preview:free run (which was mislabeled 'Hunyuan T1').
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. 39/39 calls successful.
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. 39/39 calls successful (1 dimension name retry).
18 isolated subagent evaluations via Claude Code Agent tool (no cross-scenario anchoring)
Single-pass expert analysis with full document context
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. Required max_tokens bump to 4096.
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. 1 eval failed (missing graceful_degradation).
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed.
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. 39/39 calls successful. Retested after subagent run showed 8% F2 — dimension name compliance issue was subagent-specific.
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios. 38/39 calls successful (1 timeout retry).
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios, 39/39 calls completed
OpenRouter API via lab-01 pipeline
OpenRouter API via lab-01 pipeline, 18/18 calls completed
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios; wall time inflated by free-tier rate limiting — not directly comparable to other models
OpenRouter API via lab-01 pipeline
OpenRouter API via lab-01 pipeline, structured prompts, 13 scenarios
OpenRouter API via lab-01 pipeline
OpenRouter API via lab-01 pipeline, 8/18 calls completed (heavy rate limiting)
OpenRouter API via lab-01 pipeline, 17/18 calls completed
18 isolated subagent evaluations via Claude Code Agent tool — used wrong dimension names in 14/18 responses, 1 refusal, 1 cheat (read source files). Scored with non-compliant outputs treated as all-missed.
A single deployment-safety score combining recall and precision, weighting a caught gate 4× over a false alarm. Recall alone saturates (many models catch every gate); F2 is what actually separates them. Expand any row for its recall, precision, and error counts.
A hard A-gate (Reg=A or Blast=A) that failed to fire lets a real risk through. A false alarm only adds review — so this is the number to watch after F2.
How a model gets things wrong, not how often — so it's orthogonal to rank. An even-keeled model can still miss gates and sit below a jittier one; the label describes error character, F2 does the ranking. Only sleepy (under-flags risk even when precision looks perfect — the profile to fear most) and broken are colored as warnings.
Three new models join the board, bringing it to 28 entries. GLM 5.2 is the strongest debut since March, entering at #4 (F2 0.95) with zero false alarms and the best fingerprint match of any API model (76%). Tencent Hunyuan Hy3 — the paid GA endpoint of Hunyuan 3.0 — supersedes the earlier free-preview run that was mislabeled "Hunyuan T1", trading that run's jittery 4-false-alarm profile for an even-keeled 1 FN / 1 FP at F2 0.93. MiMo v2.5, Xiaomi's open release of the family behind the Hunter and Healer stealth models, lands mid-field at F2 0.80. This snapshot also renames the "calibrated" bias profile to "even-keeled" — same definition, clearer name.
GLM 5.2 enters at #4 with F2 0.95 — the highest debut since Gemini 2.5 Flash Lite and Qwen3 235B joined in March. Its profile is precision-perfect: 100% hard gate precision (zero false alarms) with 93% recall (one missed gate), and its 76% fingerprint match edges out Owl Alpha and MiMo v2.5 (75%) as the best severity calibration of any API model. It joins the even-keeled group, which now spans the entire top 5. The trade-off is cost: at $0.22/eval it is the most expensive API model on the board.
The earlier hy3-preview:free run was mislabeled "Hunyuan T1"; the paid GA endpoint of Hunyuan 3.0 (295B MoE) now supersedes it. On paper the GA model scores lower — F2 0.93 vs the preview's 0.95 — but the error profile flipped entirely: the preview caught every gate while firing 4 false alarms (jittery), the GA misses one gate but fires only one false alarm (even-keeled). F2's 4× recall weighting deliberately favors the preview's trigger-happy profile, so the GA ranks lower despite being better behaved — exactly the asymmetry ADR-002 documents. It is also 5.7× faster than the preview run (32 min vs 3 hours).
Hunter Alpha and Healer Alpha were previously identified as Xiaomi MiMo-V2 variants. Xiaomi's open MiMo v2.5 release now lands at F2 0.80 — above Hunter (0.74) and Healer (0.62), but well short of Owl Alpha (0.89), which remains unattributed. Its 75% fingerprint match ties MiniMax M2.7 and Owl Alpha for best-in-class below the top tier, and its 0.48 differentiation is the highest in the field — the three stakeholder personas genuinely disagree, which is the intended behavior. The noisy bias (3 FN, 3 FP) keeps it out of governance-grade territory, but at $0.04/eval it is one of the cheapest credible mid-field options.
| # | Model | F2 | A-Gate Recall | Calibration | FN |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6subagent | 100% | 100% | 89% | 0 |
| 2 | Gemini 2.5 Flash Liteapi | 99% | 100% | 60% | 0 |
| 3 | Qwen3 235Bapi | 97% | 100% | 66% | 0 |
| 4 | Tencent Hunyuan T1api | 95% | 100% | 70% | 0 |
| 5 | Claude Haiku 3.5api | 92% | 93% | 71% | 1 |
| 6 | Poolside Laguna M.1api | 92% | 93% | 72% | 1 |
| 7 | Gemini 3.1 Flash Liteapi | 92% | 93% | 71% | 1 |
| 8 | Claude Sonnet 4.6subagent | 89% | 92% | 39% | 1 |
| 9 | Claude Opus 4.6manual | 89% | 87% | 90% | 2 |
| 10 | Owl Alpha (stealth)api* | 89% | 87% | 75% | 2 |
| 11 | MiniMax M2.7api | 87% | 87% | 75% | 2 |
| 12 | Grok 4.1 Fastapi | 87% | 87% | 67% | 2 |
| 13 | Baidu CoBuddyapi | 83% | 80% | 68% | 3 |
| 14 | DeepSeek v3.2api | 82% | 80% | 61% | 3 |
| 15 | DeepSeek V4 Flashapi | 79% | 80% | 59% | 3 |
| 16 | Claude Haiku 4.5 (api)api | 79% | 80% | 58% | 3 |
| 17 | InclusionAI Ring 2.6 1Tapi | 75% | 73% | 70% | 4 |
| 18 | Hunter Alpha (1T, stealth)api | 74% | 73% | 43% | 4 |
| 19 | Poolside Laguna XS 2api | 73% | 73% | 48% | 4 |
| 20 | Qwen3.6 Plusapi | 70% | 67% | 63% | 5 |
| 21 | Healer Alpha (omni, stealth)api | 62% | 60% | 47% | 6 |
| 22 | GPT-5.4 Nanoapi | 59% | 53% | 58% | 7 |
| 23 | Arcee Trinity (free)api | 57% | 53% | 49% | 7 |
| 24 | Gemma 4 26B A4Bapi | 45% | 40% | 55% | 9 |
| 25 | Nvidia Nemotron 3 Nano Omni 30Bapi | 37% | 33% | 34% | 10 |
| 26 | Claude Haiku 4.5subagent | 8% | 7% | 6% | 14 |
* Owl Alpha (stealth): NEW
Owl Alpha — OpenRouter's third anonymous "stealth" model — enters at rank 10 with F2 0.89, tying Claude Opus 4.6 (manual) and Claude Sonnet 4.6 (subagent). It achieves 100% precision (zero false alarms) with calibrated bias, making it the best free stealth model by a wide margin over Hunter Alpha (F2 0.74) and Healer Alpha (F2 0.62). Leaderboard now has 26 models.
Owl Alpha (F2 0.89, calibrated) is the first stealth model to reach the top 10. It shares the exact F2 score with Claude Opus 4.6 manual and Claude Sonnet 4.6 subagent, though its fingerprint match (75%) falls between them (90% vs 39%). Like CoBuddy and GPT-5.4 Nano, it achieves 100% hard gate precision — zero false positives — but with much better recall (87% vs 80% vs 53%). The calibrated bias is notable: only 4 models achieve it (both Opus variants, Gemini Flash Lite, and now Owl Alpha). Provider undisclosed.
The three OpenRouter stealth models now span a 27-point F2 range: Owl Alpha (0.89), Hunter Alpha (0.74), Healer Alpha (0.62). Owl is calibrated, Hunter is noisy, Healer is sleepy — three entirely different error profiles. If these are from the same provider family (as Hunter/Healer were identified as Xiaomi MiMo-V2 variants), Owl represents a significant generation jump in regulatory reasoning capability.
| # | Model | F2 | A-Gate Recall | Calibration | FN |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6subagent | 100% | 100% | 87% | 0 |
| 2 | Gemini 2.5 Flash Liteapi | 99% | 100% | 60% | 0 |
| 3 | Qwen3 235Bapi | 97% | 100% | 66% | 0 |
| 4 | Tencent Hunyuan T1api* | 95% | 100% | 66% | 0 |
| 5 | Poolside Laguna M.1api* | 92% | 93% | 63% | 1 |
| 6 | Claude Sonnet 4.6subagent | 89% | 92% | 39% | 1 |
| 7 | Claude Opus 4.6manual | 89% | 87% | 89% | 2 |
| 8 | MiniMax M2.7api | 87% | 87% | 68% | 2 |
| 9 | Grok 4.1 Fastapi | 87% | 87% | 67% | 2 |
| 10 | Baidu CoBuddyapi* | 83% | 80% | 61% | 3 |
| 11 | DeepSeek v3.2api | 82% | 80% | 61% | 3 |
| 12 | InclusionAI Ring 2.6 1Tapi* | 75% | 73% | 64% | 4 |
| 13 | Hunter Alpha (1T, stealth)api | 74% | 73% | 43% | 4 |
| 14 | Poolside Laguna XS 2api* | 73% | 73% | 48% | 4 |
| 15 | Qwen3.6 Plusapi | 70% | 67% | 59% | 5 |
| 16 | Healer Alpha (omni, stealth)api | 62% | 60% | 49% | 6 |
| 17 | GPT-5.4 Nanoapi | 59% | 53% | 50% | 7 |
| 18 | Arcee Trinity (free)api | 57% | 53% | 48% | 7 |
| 19 | Nvidia Nemotron 3 Nano Omni 30Bapi* | 40% | 36% | 34% | 9 |
| 20 | Claude Haiku 4.5subagent | 8% | 7% | 6% | 14 |
* Tencent Hunyuan T1: NEW
* Poolside Laguna M.1: NEW
* Baidu CoBuddy: NEW
* InclusionAI Ring 2.6 1T: NEW
* Poolside Laguna XS 2: NEW
* Nvidia Nemotron 3 Nano Omni 30B: NEW
Six new models bring the leaderboard to 20 entries. Tencent Hunyuan T1 joins the perfect-recall club at rank 4 but with jittery bias (4 false positives) and extreme verbosity (~9.5k tokens/response, ~3h wall time). Poolside debuts with two models — Laguna M.1 (rank 5, 93% recall) and Laguna XS 2 (rank 14, 73% recall) — showing a clear size/quality tradeoff. Baidu CoBuddy achieves 100% precision but misses 3 gates (sleepy). Nvidia Nemotron 3 Nano Omni 30B (30B/3B activated) lands near the bottom at 36% recall. The top 3 remain unchanged: Opus subagent, Gemini Flash Lite, Qwen3 235B.
Hunyuan T1 catches every hard gate (100% recall) but fires 4 false positives — the most of any model with perfect recall. At ~9.5k tokens per response it required removing the max_tokens cap entirely, and 2 of 39 calls timed out. Wall time was ~3 hours (10,913s), though this likely reflects free-tier rate limiting rather than raw inference speed. Fingerprint match (66%) ties Qwen3 235B. The jittery bias makes it conservative — it over-flags rather than under-flags — which is preferable to sleepy for safety-critical screening, but the verbosity and timeout issues are operational concerns.
Laguna M.1 (rank 5, F2 0.92, 93% recall) is the strongest new entrant — it slots between the perfect-recall club and the Claude/MiniMax/Grok tier. Laguna XS 2 (rank 14, F2 0.73, 73% recall) performs 20 points lower but completes in just 126s vs 1,020s for M.1. Both show jittery/noisy bias (false positives >= false negatives). The M.1 result is notable: it outperforms Claude Sonnet 4.6 subagent on F2 despite being an API model without the subagent pipeline advantage.
CoBuddy achieves 100% precision (zero false positives) but only 80% recall — the inverse of Hunyuan T1. Where Hunyuan over-triggers on safe scenarios, CoBuddy under-detects real risks. This makes CoBuddy the model of choice when false alarms are expensive but missed gates are tolerable. It joins GPT-5.4 Nano and Claude Opus manual as the only models with zero false positives, though at very different recall levels (53% vs 87% vs 80%).
Despite its 1T parameter count, Ring 2.6 lands at rank 12 with 73% recall and sleepy bias — matching Hunter Alpha (also labelled 1T) almost exactly. The fingerprint match (64%) is above average, suggesting decent dimension-level understanding, but the 4 missed gates place it firmly in the moderate tier. Parameter count alone does not predict gate detection performance.
At 30B total / 3B activated, Nemotron 3 Nano Omni is the smallest model in the benchmark. It completed only 17/18 scenarios and achieved just 36% recall with 9 missed gates — the worst performance above the broken Haiku tier. Fingerprint match (34%) is also the lowest among functional models. The "reasoning" label in its model ID did not translate to gate detection capability at this parameter scale.
With 20 models, the tiers have solidified. Perfect recall (100%): Opus subagent, Gemini Flash Lite, Qwen3 235B, Hunyuan T1. Strong (87–93%): Laguna M.1, Sonnet, Opus manual, MiniMax, Grok. Moderate (73–83%): CoBuddy, DeepSeek, Ring 2.6, Hunter, Laguna XS 2. Weak (53–67%): Qwen3.6 Plus, Healer, GPT-5.4 Nano, Arcee. Near-broken (≤40%): Nemotron 3 Nano, Haiku.
| # | Model | F2 | A-Gate Recall | Calibration | FN |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6subagent | 100% | 100% | 87% | 0 |
| 2 | Gemini 2.5 Flash Liteapi | 99% | 100% | 60% | 0 |
| 3 | Qwen3 235Bapi | 97% | 100% | 66% | 0 |
| 4 | Claude Sonnet 4.6subagent | 89% | 92% | 39% | 1 |
| 5 | Claude Opus 4.6manual | 89% | 87% | 89% | 2 |
| 6 | MiniMax M2.7api | 87% | 87% | 68% | 2 |
| 7 | Grok 4.1 Fastapi | 87% | 87% | 67% | 2 |
| 8 | DeepSeek v3.2api | 82% | 80% | 61% | 3 |
| 9 | Hunter Alpha (1T, stealth)api | 74% | 73% | 43% | 4 |
| 10 | Qwen3.6 Plusapi* | 70% | 67% | 59% | 5 |
| 11 | Healer Alpha (omni, stealth)api | 62% | 60% | 49% | 6 |
| 12 | GPT-5.4 Nanoapi | 59% | 53% | 50% | 7 |
| 13 | Arcee Trinity (free)api | 57% | 53% | 48% | 7 |
| 14 | Claude Haiku 4.5subagent* | 8% | 7% | 6% | 14 |
* Qwen3.6 Plus: NEW
* Claude Haiku 4.5: Instruction-following failure — see analysis below
Qwen3.6 Plus joins at rank 10 with 70% F2, 67% recall, sleepy bias — but the older Qwen3-235B-A22B (rank 3, 100% recall) outperforms it by 33 points. Wall time is not comparable (free-tier rate limiting inflated benchmark duration). Qwen3.5 Plus params (397B/17B) are known; 3.6 Plus params are undisclosed. DeepSeek v3.2 drops to rank 8 after a failed eval. 14 models across 4 tiers.
Qwen3.6 Plus (67% recall) significantly underperforms Qwen3-235B-A22B (100% recall) — a 33-point gap. Qwen3.6 Plus (hybrid architecture: linear attention + sparse MoE, always-on CoT, 1M context, 78.8% SWE-bench) is optimized for agentic coding but underperforms on safety gate detection. Qwen3-235B-A22B (235B total / 22B activated, 94 layers, 128 experts, 262K context, Apache 2.0 license) remains the clear choice for gate detection despite being older.
The reported 51-minute wall time is not a fair benchmark — the free-tier endpoint was heavily rate-limited with exponential backoff, inflating total time. Actual per-call latency (~89s on the paid endpoint) suggests it is fast, but a clean paid-tier run is needed to confirm. What is confirmed: its 67% recall / 91% precision reflects a "sleepy" bias — it errs toward under-flagging risks. For screening where false alarms are costly, the precision is appealing, but the 33% miss rate is a liability for safety-critical gate detection.
DeepSeek v3.2 completed only 17/18 scenarios, missing the graceful_degradation scenario entirely. This pulled its recall down to 80% from the previous 87% (estimated from imputed data). The benchmark now reflects the real-world completion rate — models that fail to produce valid outputs are scored accordingly.
The leaderboard continues to show clear performance bands. Top tier (100% recall): Opus subagent, Gemini, Qwen3 235B. Strong tier (87–92% recall): Sonnet, Opus manual, MiniMax, Grok. Moderate tier (53–73% recall): DeepSeek, Hunter, Qwen3.6 Plus, Healer, GPT-5.4 Nano, Arcee. Broken: Haiku. Qwen3.6 Plus lands in the moderate tier alongside other sleepy models.
| # | Model | F2 | A-Gate Recall | Calibration | FN |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6subagent | 100% | 100% | 87% | 0 |
| 2 | Gemini 2.5 Flash Liteapi | 99% | 100% | 60% | 0 |
| 3 | Qwen3 235Bapi | 97% | 100% | 66% | 0 |
| 4 | Claude Opus 4.6manual | 89% | 87% | 89% | 2 |
| 5 | Claude Sonnet 4.6subagent | 89% | 92% | 39% | 1 |
| 6 | Grok 4.1 Fastapi | 87% | 87% | 67% | 2 |
| 7 | MiniMax M2.7api | 87% | 87% | 68% | 2 |
| 8 | DeepSeek v3.2api | 82% | 80% | 61% | 3 |
| 9 | Hunter Alpha (Xiaomi MiMo-V2-Pro)api | 74% | 73% | 43% | 4 |
| 10 | Healer Alpha (Xiaomi MiMo-V2-Omni)api | 62% | 60% | 49% | 6 |
| 11 | GPT-5.4 Nanoapi | 59% | 53% | 50% | 7 |
| 12 | Arcee Trinityapi | 57% | 53% | 48% | 7 |
| 13 | Claude Haiku 4.5subagent* | 8% | 7% | 6% | 14 |
* Claude Haiku 4.5: Instruction-following failure — see analysis below
Six new API models join the leaderboard: Gemini 2.5 Flash Lite (recall 100%), Qwen3 235B (recall 100%), MiniMax M2.7 (recall 87%), Grok 4.1 Fast (recall 87%), DeepSeek v3.2 (recall 80%), and GPT-5.4 Nano (recall 53%). Gemini and Qwen3 achieve perfect hard gate recall — matching Opus subagent on the most important metric. GPT-5.4 Nano is the first OpenAI model on the leaderboard: it never false-alarms (100% precision) but misses half the hard gates (53% recall, sleepy bias), ranking just above Arcee Trinity. It is the second-fastest model at 102 seconds and among the cheapest at $0.03/eval, but its sleepy bias makes it unsuitable for safety-critical use. The field has grown to 13 models across 4 performance tiers.
Both Gemini 2.5 Flash Lite and Qwen3 235B achieve 100% hard gate recall — catching every safety-critical gate across all 18 evaluations. This matches Opus subagent performance on the most important metric. Gemini has 1 false positive, Qwen3 has 2. These are the first API-accessible models to demonstrate that perfect gate detection is achievable without subagent isolation or Opus-tier reasoning.
MiniMax M2.7 ties Grok 4.1 Fast on recall and precision (both 87%/87%) and slightly outperforms it on calibration (68% vs 67%). However, it required a max_tokens bump to 4096 (from the default 1024) to produce complete outputs, and its wall time of 1227 seconds (20+ minutes) is the slowest in the field — over twice as long as Qwen3 (613s) and 17x slower than Gemini (71s). For batch evaluations this may not matter, but for interactive use the latency is significant.
Gemini 2.5 Flash Lite completes the benchmark in 71 seconds. MiniMax M2.7 takes 1227 seconds. Between them: Healer Alpha (455s), Grok (503s), Arcee (505s), Qwen3 (613s), DeepSeek (991s), and Hunter Alpha (1185s). Speed does not correlate with accuracy — Gemini is both the fastest and the most accurate API model. Organizations choosing between models of similar accuracy can now factor in throughput.
The anonymous "stealth" models on OpenRouter have been identified as Xiaomi's MiMo-V2 series. Hunter Alpha is MiMo-V2-Pro, Xiaomi's flagship 1T-parameter MoE model (42B activated) designed as an agent brain. Healer Alpha is MiMo-V2-Omni, the multimodal variant. Hunter Alpha was previously the community's most-guessed candidate for DeepSeek V4. On ARA Eval, Hunter scores 73% recall (moderate) and Healer scores 60% (weak) — both well below the top API models despite Hunter's strong performance on coding benchmarks.
MiniMax M2.7 (68%), Grok (67%), and Qwen3 (66%) now cluster at 66–68% calibration — a meaningful improvement over the previous API cohort (43–49%) but still far from Opus manual at 89%. The gap has narrowed from ~40 points to ~21 points. Getting binary gate detection right is now solved by multiple models; severity calibration across 7 dimensions remains the frontier.
The first OpenAI model on the leaderboard scores 53% recall with 100% precision — it never raises a false alarm, but misses 7 of 15 hard gates. At 102 seconds and $0.03/eval it is the second-fastest and among the cheapest models tested. Its "sleepy" bias makes it the mirror image of Sonnet's "jittery" bias: Nano under-detects risk while Sonnet over-triggers. For screening where false alarms are costly, Nano's precision is appealing — but for safety-critical gate detection, its 53% recall is disqualifying.
The leaderboard now has clear performance bands. Top tier (100% recall): Opus subagent, Gemini, Qwen3. Strong tier (87–92% recall): Opus manual, Sonnet, Grok, MiniMax. Moderate tier (53–80% recall): DeepSeek, Hunter, Healer, GPT-5.4 Nano, Arcee. Broken: Haiku. Organizations can make informed trade-offs between gate accuracy, calibration quality, speed, and API availability.
| # | Model | F2 | A-Gate Recall | Calibration | FN |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6subagent | 100% | 100% | 87% | 0 |
| 2 | Claude Opus 4.6manual | 89% | 87% | 89% | 2 |
| 3 | Hunter Alphaapi | 74% | 73% | 43% | 4 |
| 4 | Healer Alphaapi | 62% | 60% | 49% | 6 |
| 5 | Arcee Trinityapi | 57% | 53% | 48% | 7 |
Models are ranked by F2 score, which weights recall 4× over precision — because a missed risk gate is more dangerous than a false alarm. The F2 ordering is corroborated by hard_gate_recall and false_negative counts independently, making the ranking robust across metrics. There is a sharp performance cliff between Opus-tier models (F2 ≥ 0.89) and the rest (F2 ≤ 0.74), with no middle ground. The non-Opus models cluster near coin-flip on dimension-level accuracy (0.43–0.49), suggesting the current benchmark separates "gets it" from "guessing" without much gradation. All API models currently complete 18/18 evaluations reliably and are free to use — so the comparison is purely on classification quality.
Claude Opus 4.6 achieves perfect gate recall (1.0) when run as 18 isolated subagent evaluations, but drops to 0.87 in single-pass manual mode — missing 2 gates. The manual approach slightly outperforms on dimension-level calibration (0.89 vs 0.87), suggesting cross-scenario context helps severity assessment even as it introduces gate-detection noise.
Hunter Alpha catches 3 more gates than the next-best API model (4 FN vs 7), giving it the strongest safety-critical performance in the free tier. At 74% F2, it is the recommended default for anyone evaluating real scenarios — it misses fewer dangerous gates. If you need a model that errs on the side of catching risk, Hunter is the clear choice among currently-free options.
Arcee has the highest precision among API models (80%) and the most persona differentiation (69%), which makes results feel more interesting — the personalities disagree more. But the subagent results showed that high differentiation can be noise, not insight: Claude at 31% differentiation only disagreed where it should, and scored perfect on gating. Arcee's real strength is as a stable, permanently-free baseline — useful as a fallback when stealth models inevitably disappear.
Healer sits between Hunter and Arcee on most metrics but leads on none. Worse recall than Hunter (60% vs 73%), worse precision than Arcee (75% vs 80%), and similar dimension match to both (~0.49). Currently free like the other stealth models, but offers no differentiated advantage.
Lower-performing models show higher persona differentiation (0.60–0.69) compared to Opus (0.26–0.31). Rather than indicating genuine perspective diversity, this likely reflects output variance — the models are inconsistent across personas, not genuinely surfacing different viewpoints. Claude's lower differentiation was concentrated on borderline regulatory calls where CO/CRO disagreement is expected.
At 43–49% dimension match, all non-Opus API models are near coin-flip on severity calibration across 7 risk dimensions. These models are valuable for learning the framework, running demos, and generating discussion — but nobody should make actual governance decisions based on their output alone. The benchmark is currently best used as a teaching tool.