Binary Attack Metrics Hide Catastrophic Agent Failures — New Severity Scale Exposes the Ga

A defense that reported a zero attack-success rate still let an externally visible cross-scope leak slip through an unfiltered tool [^claim_1836]. That finding comes from a new paper on severity grading for tool-using AI agents, and it exposes a blind spot that hits home for crypto protocols running autonomous agents — MEV searchers, cross-chain relayers, DAO treasury bots, AI-powered smart-contract auditors. Binary attack-success rate throws away the information a defender actually needs: how harmful the resulting action was [^claim_1838]. A compromised agent that leaks a private key is catastrophically different from one that reads a public log, but binary metrics treat them the same.

The paper introduces an action-graded harm rubric. It scores an agent’s tool-call trajectory on a seven-level ordinal scale (L0 to L6) based on whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege [^claim_1834]. The scale is computed two ways: a deterministic oracle that reads the trajectory and the attacker’s stated goal, and a panel of three frontier language-model judges that read a tag-free account of the same trajectory [^claim_1835]. Across four victim models and two defenses on the AgentDojo workspace suite, severity grading exposes three cases the binary metric hides [^claim_1836]. The judge panel reproduces the oracle with high ordinal agreement (Krippendorff’s alpha = 0.91) but shares systematic blind spots — most notably, it fails to recognize escalation chains [^claim_1837].

For crypto, the implications are immediate. Protocols like EigenLayer’s AVS or Olas’ autonomous agent marketplaces could adopt the L0–L6 scale to classify incident severity in slashing conditions or insurance claims. That replaces today’s binary “was the agent slashed?” with a graduated penalty tied to actual harm. The rubric is trace-grounded — it’s applied to actual actions recorded in existing red-team logs, not a harm taxonomy or synthetic simulation [^claim_1839]. Security firms like OpenZeppelin, Trail of Bits, or Forta Network can retrofit the scale onto existing alert pipelines: L4+ triggers automatic key rotation or transaction pausing. The LLM judge’s blind spot on escalation chains argues for on-chain or TEE-enforced action logging with deterministic severity computation (the oracle approach) rather than post-hoc LLM judging, especially for agents handling high-value keys or signing operations.

All code, prompts, and per-episode logs are released [^claim_1840], so the crypto security community can fork and adapt the rubric for on-chain agent environments — adding crypto-specific dimensions like value-at-risk in ETH or number of affected user positions. The binary era for agent incident response is over. The question now is which protocols will adopt a graduated severity scale first — and which will learn the hard way that zero attack-success rate does not mean zero harm.