Researchers have developed a novel approach to improve how AI-powered coding assistants fix software bugs by teaching them to generate better tests and learn from the results. The new system, called ExecCritic, aims to make automated code repair more reliable by separating the tasks of writing tests and fixing code, then using feedback from test results to guide improvements. This matters because many AI coding tools struggle when their own tests are flawed, leading to false confidence in buggy fixes. ExecCritic’s approach helps ensure tests accurately capture the intended behavior, leading to more trustworthy bug fixes.
Key Takeaways
- ExecCritic splits the process into two AI agents: one generates tests, the other fixes code based on test feedback.
- The test agent learns to create behaviorally valid tests that can distinguish correct from incorrect code patches.
- The repair agent uses test execution results to guide code revisions without altering the tests themselves.
- On a benchmark of software repair tasks, ExecCritic improved the success rate of correctly fixed bugs by over 11 percentage points compared to methods without better testing.
At the heart of ExecCritic is a “test–verify–revise” cycle designed to improve automated code repair. Traditional AI coding agents often generate both a patch (a fix) and tests to check that patch. But if both the patch and tests are produced together by the same process, mistakes in the tests can falsely validate faulty patches, giving a misleading sense of correctness.
To address this, ExecCritic uses two separate AI agents, both based on a large language model called Qwen-3.5-35B-A3B. The first, called the Test agent, independently creates tests that are “repository-native,” meaning they fit naturally with the existing codebase and clearly define the expected behavior. These tests go through a “fail-closed harness” — a system that filters out any tests that do not reliably fail when the code is wrong, ensuring only high-quality tests are used.
The second, the Repair agent, takes these vetted tests and uses the feedback from running them to guide its code fixes. Crucially, it does not change the tests themselves, which prevents the kind of circular errors that can occur when tests and fixes are generated together. Both agents are trained separately using reinforcement learning — a method where the AI learns by trial and error, receiving rewards for producing better results.
Testing ExecCritic on the SWE-bench Verified dataset, which contains various real-world software repair challenges, showed that the quality of tests significantly impacts repair success. When using tests from a baseline Test agent, the repair success actually dropped compared to not using tests at all. However, when using tests from a more advanced GPT-5.6-sol model, success rates improved noticeably. By further training the Qwen-based Test agent with role-specific fine-tuning, researchers boosted its ability to generate valid tests, which in turn helped the Repair agent raise bug-fixing success substantially — from 22.2% to 62.2% when improving the Test agent alone, and up to 72.6% when both agents were post-trained and combined.
This research highlights the importance of test quality in AI-driven code repair and introduces a promising framework for improving it through specialized roles and feedback loops. By teaching AI agents not only to fix code but also to critically test their own fixes, ExecCritic moves closer to more dependable automated software maintenance.
Looking ahead, the researchers have made the ExecCritic code publicly available, inviting further exploration and development. While this approach shows clear benefits on benchmark tasks, real-world software projects often involve even more complexity. Future work may explore scaling these methods to larger codebases, integrating with human developers, and refining the agents’ abilities to understand nuanced software requirements. Nonetheless, ExecCritic represents a meaningful step toward smarter, self-improving AI coding assistants that can better support software development workflows.
Based on research published on arXiv by Leitian Tao, Baolin Peng, Haorui Wang et al..