Binary Attack Metrics Hide AI Agent Harm: A Seven-Level Severity Scale for Crypto

Binary attack-success rates are the default metric for evaluating AI agent security, but they systematically discard the information a defender most needs: how harmful the resulting action actually was [^claim_1801]. A new paper demonstrates this failure empirically—a defense can report a 0% attack-success rate while still permitting an externally visible cross-scope leak through an unfiltered tool [^claim_1799]. For crypto agents that control wallets, execute DeFi trades, or manage on-chain governance, this blind spot is not theoretical. A binary pass/fail on an agent controlling a multi-sig could mask a partial compromise that leaks a price oracle signal or executes a reversible but damaging trade.

The paper introduces a seven-level ordinal scale (L0 to L6) that scores an agent’s tool-call trajectory according to three criteria: whether the action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege [^claim_1797][^claim_1803]. This rubric directly maps to on-chain actions. A reversible swap on a DEX is L1–L2; a cross-contract call that modifies another user’s position is L3–L4; a privilege escalation that changes contract ownership is L5–L6. The scale is computed two ways: a deterministic oracle that reads the trajectory and the attacker’s stated goal, and a panel of three frontier language-model judges that read a tag-free account of the same trajectory [^claim_1798]. The judge panel reproduces the oracle with high ordinal agreement (Krippendorff’s alpha = 0.91) but shares systematic blind spots, most notably a failure to recognize escalation chains [^claim_1800].

For crypto, the implications are concrete. Any AI agent managing on-chain assets—from autonomous MEV bots to intent-solver agents on EigenLayer AVSs—currently evaluated with binary metrics could be hiding partial compromises. The severity scale provides a structured way to grade actual tool-call trajectories rather than just whether an injection ‘worked.’ ZK-rollup sequencers and TEE-based agent runtimes (e.g., Phala Network, Oasis Sapphire) that log tool-call trajectories could use this rubric as a post-hoc severity classifier. The blind spot on escalation chains is critical for on-chain governance agents: an attacker who chains a series of seemingly harmless proposals (each reversible individually) could escalate to contract ownership without the monitor flagging it.

The contribution is a reusable, trace-grounded severity instrument applied to actual actions recorded in existing red-team logs, with all code, prompts, and per-episode logs released [^claim_1802]. For projects like Autonolas, Fetch.ai, or EigenLayer’s AVS-based agent networks, this is a drop-in evaluation tool for their agent security testing pipelines. The shift from binary audit pass/fail to severity-scored vulnerability classification—mirroring Immunefi’s severity tiers but applied to AI agent actions—is now directly applicable.

Bottom line: Binary attack-success metrics are insufficient for tool-using AI agents in crypto. The severity scale exposes hidden harms, and its released implementation allows immediate adoption. Watch for projects integrating this rubric into their agent monitoring and circuit-breaker logic.