What Is Agentic Risk Assessment?

The cost of building software is collapsing. Tools like OpenClaw generate entire applications from prompts. A 70-person manufacturer in Hong Kong replaced its entire SaaS stack with AI-generated applications consuming 250 million tokens a day. As Forrester's Frederic Giron observed in “When Code Is Free, What's Left to Sell?” , AI-assisted builds are proliferating faster than governance frameworks can keep up.

The result: enterprises feel intense pressure to deploy agent systems but have no structured way to assess the risk. They can't answer basic questions: Will this agent flag a compliance violation before acting on it? Will it respect data access boundaries? Will it escalate when it should, or will it silently proceed?

The stakes are real. Knight Capital lost $440 million in 45 minutes when a deployment error on a single server cascaded through algorithmic trading. UnitedHealth's nH Predict algorithm denied rehabilitation care with a 90% override rate by human reviewers. AIA was fined HK$23 million after its AML screening algorithm missed politically exposed persons for 6 years.

Agentic risk assessment gives organizations a way to catch these failures before deployment — not after an incident.

ARA Eval produces a risk fingerprint — a 7-character classification like C-B-A-D-D-B-C where each letter represents one dimension. Level A is highest risk, Level D is lowest.

A risk gate is a point in a decision where an AI agent should pause, flag, or escalate rather than proceeding autonomously. Gates are the moments where judgment matters — where the agent's next action could create compliance exposure, financial loss, or safety harm.

ARA Eval applies gating rules deterministically in code, not by the LLM. This is a deliberate architectural choice: the LLM classifies risk levels, but the gating logic is auditable, replicable, and enforceable. You can swap models, change rubrics, or adjust personas — the gates remain constant.

Hard gates

Hard gates override everything else. If triggered, autonomy is not permitted regardless of how the other dimensions score.

• Regulatory Exposure = A — autonomy not permitted; human-in-loop required
• Failure Blast Radius = A — human oversight required; supervised autonomy may be possible with monitoring

Soft gates

Soft gates require documented risk acceptance from the appropriate authority, but don't automatically block deployment.

• Any dimension = A (other than the hard gates) — requires documented risk acceptance
• All dimensions ≥ C — strong candidate for full autonomy

Readiness classifications

Every evaluation produces one of three verdicts:

ARA Eval tests AI agents against 6 core enterprise scenarios drawn from Hong Kong financial services — banking, insurance, capital markets — with regulatory context from HKMA, SFC, PCPD, and PIPL. Each scenario is evaluated from 3 stakeholder perspectives:

This produces 18 evaluations(6 scenarios × 3 perspectives), each generating a risk fingerprint. Where perspectives disagree, that disagreement reveals where organizational alignment is needed before deployment.

Results are scored against human-authored reference fingerprints — expert-written assessments that define what a correct risk evaluation looks like. The primary ranking metric is F2 score, which weights recall 4x over precision: catching dangerous gates matters more than avoiding false alarms.

Each scenario is drawn from a real or realistic enterprise situation, calibrated to test different aspects of risk judgment. Several reference real incidents.

An additional 7 backup scenarios are available covering fraud detection, market surveillance, credit scoring, AML screening, underwriting, trade surveillance, and robo-advisory — all with reference fingerprints based on real regulatory incidents.

Most AI benchmarks test capability — can the model answer questions, write code, pass exams? ARA Eval tests judgment — does the model know when to stop, escalate, or flag a risk?

Three architectural decisions set the framework apart:

• Deterministic gating — the LLM classifies risk levels, but gating rules are applied in code. This separates probabilistic classification from enforceable policy. The LLM cannot talk itself out of its own ratings.
• Multi-perspective evaluation — three stakeholder personas (Compliance, Revenue, Operations) evaluate the same scenario. Where they disagree is where your organization needs alignment before deployment.
• Recursive pedagogy — the framework uses AI to evaluate AI autonomy, which teaches skepticism about automated judgment. Lab 03 shows LLMs disagreeing with themselves. Lab 02 shows sensitivity to framing. The lesson: healthy distrust of automated risk assessment, learned by doing it.

The ARA Eval leaderboard shows a sharp performance cliff. Claude Opus 4.6 achieves perfect gate recall when run as 18 isolated subagent evaluations — catching every gate that should fire. All other tested models score between 53% and 73% gate recall, with dimension-level accuracy near coin-flip (43–49%).

This means no current free API model is ready for real governance decisions on its own. But that's the point: the benchmark exists to make that gap visible, measurable, and trackable as models improve.

ARA Eval is open source. The default judge model (Arcee Trinity) is free via OpenRouter. A full core evaluation runs 18 calls and takes about 5 minutes.