Training artificial intelligence (AI) to write and fix computer code is a complex challenge that requires diverse, realistic tasks for the AI to practice on. A newly published research paper introduces CodeMidas, a novel system that creates thousands of coding challenges directly from existing software projects, without relying on extra information like bug reports or update histories. This approach could significantly expand the variety and scale of tasks available for teaching coding AI, helping these agents become better at understanding and generating code across many programming languages and domains.
Key Takeaways
- CodeMidas automatically transforms source code from over 3,000 open-source projects into 5,545 unique coding tasks spanning 23 programming languages and 15 technical fields.
- The system uses AI-driven exploration to understand code functionality, generate tests, and verify tasks by running code and solutions repeatedly.
- Training a coding agent on these tasks led to measurable improvements on five different coding benchmarks, including fixing bugs and building entire programs.
- More high-quality training tasks resulted in better AI performance, demonstrating the value of scalable, diverse coding challenges.
The core idea behind CodeMidas is to harness the wealth of existing software code as a playground for AI learning. Traditionally, creating tasks for training coding agents has depended on supplemental development artifacts like issue trackers or commit histories, which can be inconsistent or unavailable. Instead, CodeMidas uses only the source code itself as its input. It assigns “agentic compute” — meaning it employs AI agents to actively engage with the code at every step of the environment creation process.
First, the AI explores the implemented functions to figure out what the code is supposed to do, essentially generating behavioral specifications. Next, it constructs tests that are grounded in running the original code, ensuring that the tasks are meaningful and executable. Finally, it performs repeated execution checks and solution rollouts to validate and filter the tasks, keeping only those that are reliable for training. This pipeline results in a large, diverse dataset of coding challenges without manual task design.
The researchers then trained an advanced coding agent, called MiMo-V2.5, using a reinforcement learning method known as GRPO on this dataset. Reinforcement learning is a type of AI training where agents learn by trial and error to maximize rewards—in this case, successfully completing coding tasks. The agent showed improved performance across multiple benchmarks, including DeepSWE for bug fixing, ProgramBench for whole-program synthesis, and Terminal-Bench for command-line tasks. Analysis of the agent’s behavior revealed that it explored codebases more thoroughly and verified its work more diversely, indicating more sophisticated problem-solving strategies.
This research highlights the potential of using source code itself as a scalable foundation for creating training environments for coding AI. By unlocking a vast array of tasks from existing software, CodeMidas could accelerate the development of more capable and versatile coding agents. These improvements may eventually aid software developers by automating routine coding tasks, debugging, or even generating complex programs. Future work could explore integrating this approach with other AI techniques or expanding the range of programming languages and domains further.
Based on research published on arXiv by Bowen Ye, Lei Li, Shicheng Li et al..