As artificial intelligence becomes a bigger part of software development, researchers are eager to understand how well AI systems can grasp not just what code looks like, but how it actually behaves when it runs. A newly published study tackles this question by testing large language models (LLMs) on their ability to reason about the dynamic, or runtime, behavior of entire software projects. This is important because understanding the flow of a program during execution is key to debugging, optimization, and writing reliable code—skills that AI tools are increasingly expected to assist with.
Key Takeaways
- The researchers created SWE-Flux, a new benchmark with 480 questions that require reasoning about how real Python repositories behave when executed.
- Unlike previous tests, SWE-Flux uses automatically collected “ground truth” answers from actual program runs, avoiding human bias or AI-generated evaluations.
- Five popular large language models were tested, but the best achieved only 37% accuracy, showing this remains a tough challenge.
- Models performed better on simpler, localized tasks like understanding loops or exceptions within a single function, but struggled with complex data flow and reasoning across multiple functions or tests.
The study focuses on “dynamic execution reasoning,” which means understanding what happens when code runs, rather than just analyzing the code’s text or structure. Previous benchmarks mostly tested models on small code snippets or static questions, but real-world software involves many interacting parts. To address this, the researchers developed SWE-Flux, a benchmark built from 12 real Python projects, each with multiple tests that exercise different parts of the code.
To create reliable answers for their benchmark, the team used an “oracle-harvesting” approach. This means they ran the software’s own tests with instrumentation—special monitoring tools—to record exactly what happens during execution. These recorded behaviors form the “gold standard” answers to the benchmark questions. This method avoids relying on human-written answers or evaluations generated by AI models themselves, which can introduce errors or bias.
The questions cover a range of concepts important to understanding program behavior, such as control flow (the order in which instructions run), loops, program state (the values stored during execution), data flow (how data moves through the program), exceptions (errors and how they are handled), and program invariants (properties that remain true throughout execution). Some questions focus on a single test case, while others require aggregating information across multiple tests, increasing difficulty.
When five leading large language models were evaluated on SWE-Flux, the results showed that even the best models struggled, achieving just 37% accuracy overall. The models were relatively good at reasoning about localized elements like simple loops, exceptions, and invariants within a single function. However, they had difficulty with more complex tasks such as tracking data as it flows between different parts of the program, reasoning about the precise state of the program at various points, and combining information across multiple tests.
Additionally, the researchers demonstrated that their oracle-harvesting pipeline can create new, more challenging benchmark variants by slightly changing inputs and rerunning tests. Nearly 90% of these variants were valid and proved even harder for the AI models, suggesting a path toward continuously improving benchmarks that push AI’s reasoning abilities further.
This research highlights that while large language models have made impressive strides in understanding code, their ability to reason about how code behaves during execution is still limited. For developers and AI tool creators, SWE-Flux offers a valuable resource to measure progress and identify areas where AI needs improvement. Looking ahead, advances in dynamic reasoning could lead to smarter coding assistants that better predict bugs, optimize performance, and help programmers understand complex software systems more effectively.
Based on research published on arXiv by Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al..