TL;DR
- Accuracy: The system reached an 88.6% cumulative exact hit rate across 35 real incidents.
- Noise Control: Multi‑round deduplication and a multi‑stage validator reduce false positives and improve clarity.
- Practical Use: The tool supports auditors and developers with structured detection suited for real‑world workflows.
The release of the AI Auditor marks a major step in how CertiK approaches automated blockchain security. After months of internal use, the system is now open to developers seeking reliable detection without the noise that often slows real audits. Its performance across real incidents shows how far the tool has evolved from its earliest versions.
— CertiK (@CertiK) April 7, 2026
Real‑World Testing Shows Strong Accuracy
The team evaluated the system against 35 verified security incidents from 2026, all rooted in Solidity code. It delivered 28 exact hits on the first run and reached 31 after a second pass, producing an 88.6% cumulative exact hit rate. These results reflect a design focused on identifying issues that matter rather than generating long lists of alerts. The goal has always been to help teams surface meaningful risks earlier in the development cycle.
Internal Feedback Drove Major Improvements
The tool began as an internal resource for CertiK auditors, who quickly pointed out that early versions created too much noise. Duplicate findings, weak context, and false positives made triage harder, not easier. This feedback shifted the product direction. Instead of chasing raw detection numbers, the team balanced detection with strict noise reduction. Over the following year, they tested multiple agent setups, workflows, and models to find combinations that consistently performed well across real cases.
Multi‑Scanner Architecture Reduces Blind Spots
One key lesson was that no single scanner can cover every vulnerability type. Different approaches excel in different areas, so the system combines several scanners whose strengths complement each other. Benchmarks showed that ensemble coverage outperformed any individual method. To prevent overload, the system applies multi‑round deduplication and a multi‑stage validator that checks structural correctness, semantic relevance, and exploitability before producing a final verdict.
Built To Support Auditors and Developers
The system’s evolution has been shaped by continuous auditor involvement. Their expertise guides benchmark design, knowledge base updates, and validator calibration. The tool is not meant to replace human judgment but to support pre‑deployment review, upgrade analysis, triage, and verification. For CertiK, releasing it publicly is the next step in refining a system built through real audit practice.
