As computer chips grow ever more complex, ensuring they work correctly before manufacturing is crucial. Engineers rely on automated test setups called “scoreboards” to verify that a chip’s behavior matches its intended design. However, these scoreboards only check specific aspects and can miss subtle bugs that still violate the overall chip specification. A newly published research paper introduces a novel approach using large language models (LLMs) to cross-check test results against the full specification, helping to uncover hidden hardware bugs that standard methods overlook.
Key Takeaways
- Traditional UVM scoreboards confirm designs match reference models on tested behaviors but can miss bugs that violate the full specification—these are called “scoreboard escapes.”
- The new framework, “Verify the Verifier,” uses LLMs to analyze passing test traces against the specification, looking for inconsistencies that indicate hidden bugs.
- Out of 564 intentionally inserted bugs in four OpenTitan hardware modules, 68 were flagged as candidate scoreboard escapes; 41 of these were confirmed as actual hidden bugs.
- This method demonstrates the potential of AI-powered cross-checking to improve hardware verification beyond traditional testing frameworks.
In chip design, verification engineers use Universal Verification Methodology (UVM) scoreboards to compare the outputs of a design under test against a trusted reference model. Passing these regression tests typically suggests the design behaves correctly. However, the scoreboard only checks certain outputs and scenarios defined by the testbench, not the entire chip specification. This means some bugs—called scoreboard escapes—can slip through if they do not cause mismatches detectable by the scoreboard but still violate the design’s intended functionality.
The researchers behind the paper developed “Verify the Verifier,” a framework that leverages large language models—advanced AI systems trained on vast amounts of text and code—to perform what they call Anomaly Cross-Checking. Essentially, after a testbench passes, their system uses the LLM to analyze the execution traces (detailed logs of the chip’s internal signals during testing) in light of the formal specification. The LLM looks for discrepancies between what the design does and what the specification requires, even when the scoreboard sees no problem.
To evaluate their approach, the team inserted 564 known bugs into four different OpenTitan hardware IP (intellectual property) blocks, a set of open-source chip components used in security-focused designs. Among these, 68 bugs were “candidate scoreboard escapes” — meaning they were exercised during testing, passed all scoreboard checks, but were potentially violating the specification. The framework then applied assertion checks—formal statements that must hold true for correct behavior—and confirmed 41 of these as genuine scoreboard escapes, effectively catching bugs that would have otherwise remained hidden.
This research highlights the limitations of relying solely on scoreboards for hardware verification and shows how integrating AI techniques can provide a deeper level of assurance. By grounding the LLM’s analysis in the formal specification, the approach avoids false alarms and focuses on meaningful inconsistencies.
Looking forward, this method could be integrated into existing verification workflows to complement traditional testing and formal verification tools. As chip designs become more complex and verification challenges grow, AI-powered cross-checking like “Verify the Verifier” offers a promising avenue to enhance the detection of subtle bugs before manufacturing, potentially saving time and reducing costly errors.
Based on research published on arXiv by Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi et al..
