Anthropic: Claude Automates Formal Software Verification

Anthropic has announced that its Claude model can now autonomously generate formal proofs to verify the safety of existing software code. In a project conducted by its Frontier Red Team in partnership with the firm Theorem, the AI system produced approximately 180 million lines of mathematical proof code in a matter of weeks. The effort focused on analyzing 123 open-source libraries drawn from Google's OSS-Fuzz security project, with the stated goal of mathematically confirming their correctness.

The process involved Claude translating crash test scenarios into mathematical problems and analyzing compiled machine code to produce proofs, which are then automatically checked by an independent verification program called Rocq. Anthropic describes this as a fully autonomous proof generation system, though it notes that the underlying tools and formal foundations were constructed by human experts at Theorem. Rather than creating new software, the model's function was to verify the safety of code that already exists.

While generating these proofs, Claude identified and subsequently fixed hundreds of bugs that conventional testing had not detected. According to the researchers, many of these flaws only manifest under highly specific conditions, such as handling unusually large input files or a precise sequence of access requests. In one instance involving a PDF reader, the system uncovered as many security bugs in a single week as are typically reported for that software over an entire year. In another case, a logical error that had remained undetected for more than a decade was discovered.

However, Anthropic has acknowledged significant limitations to this method. The company states that a proof of 'formally verified' does not equate to 'error-free,' as the mathematical guarantees only hold under strict, defined parameters. These conditions include a specific processor architecture, the execution of single-threaded programs, and input data that remains under 64 MB. In practical use, these libraries are deployed across various hardware configurations and operating environments that may not align with these constraints.

The computational expense of this verification process is also substantial, with individual proofs consuming up to hundreds of millions of tokens. Anthropic did not report the total costs associated with the project. Despite these constraints, the company suggests that if AI can verify its own code, developers may require less time for manual review, potentially reducing costs across the entire software development process. Anthropic also indicated that proof generation is becoming more efficient, noting that its newer Opus 5.5 model can solve half the proofs with about 40 percent fewer tokens than its predecessor.

Anthropic expects that the use of AI for formal verification will soon become as routine as standard software testing. The company's partner, Theorem, has outlined a roadmap that targets critical infrastructure components for future verification, including glibc, OpenSSL, and cURL. The initiative addresses a core challenge regarding AI-generated software by attempting to mathematically guarantee its safety under specified conditions.