LLM agents miss one in three Rust vulns—RustMizan benchmark exposes the gap

Current vulnerability benchmarks for Rust code are built on shaky ground: they use small, non-compilable snippets, focus only on binary classification, and ignore the risk that public datasets already appear in model training corpora [^claim_1563]. RustMizan, a new framework from academic researchers, directly addresses each of these shortcomings. It provides compilable code variants at the crate, file, and function levels, annotated for binary vulnerability detection, CWE classification, and function- and line-level localization [^claim_1564]. Critically, it also includes a paired mutation framework that generates semantics-preserving code mutants, enabling rigorous contamination testing and adversarial robustness probing [^claim_1565].

The results are sobering for anyone betting on LLM agents to automate Rust security auditing. Across four frontier models operating in an agentic setup with command-line access, binary classification accuracy sits in the 56–65% range [^claim_1566]. That means roughly one in three vulnerabilities is missed entirely. The picture worsens at finer granularity: line-level localization F1 stays near 20% [^claim_1567]. Even when the model correctly flags a file as vulnerable, it pinpoints the exact vulnerable line only one-fifth of the time. And when adversaries apply semantics-preserving mutations — reordering statements, renaming variables, splitting functions — line F1 drops by about 27% from the already-low baseline [^claim_1568].

For crypto, these numbers carry direct consequences. Smart-contract auditors working with Rust-based chains — Solana, NEAR, Polkadot’s ink!, Cosmos — have been benchmarking LLM tools on non-compilable snippets that may already be in the training data. RustMizan’s compilable, contamination-aware design reveals that current LLM agents cannot reliably detect vulnerabilities in Rust smart contracts, let alone localize them to the exact line. A 56–65% binary accuracy means any unsupervised pre-screening pipeline would miss ~35–44% of vulnerabilities. A 20% line F1 means automated patch generation or exploit-path pinpointing is essentially non-functional without heavy human oversight. And the adversarial robustness gap — a 27% drop in line F1 under semantics-preserving mutations — means that attackers who know an audit firm uses LLM agents can trivially obfuscate vulnerabilities to evade detection entirely.

For crypto security teams, the immediate takeaway is that current frontier models are not ready for unsupervised audit assistance on Rust smart contracts. The framework’s granular annotations — crate, file, function, line, plus CWE classification — allow protocol teams to identify which vulnerability classes (e.g., CWE-682: incorrect calculation, CWE-862: missing authorization) their LLM-based tooling systematically misses, and to build targeted fine-tuning datasets without data-contamination confounds. Watch for follow-up work that uses RustMizan to train specialized Rust-security models, and for adversarial robustness benchmarks that probe whether LLM-based audit pipelines can withstand semantics-preserving attacks.