Researchers have developed a new method to improve how large language models (LLMs) tackle real-world software issues by carefully selecting better training examples rather than simply using more data. This approach, called SWE-Prime, filters out unnecessary or unhelpful parts of problem-solving demonstrations, helping AI models learn more effectively and perform better on software tasks. The findings, published in a recent research paper, suggest that quality matters more than quantity when teaching AI to fix software problems.
Key Takeaways
- SWE-Prime uses a two-stage filtering process to select only the most useful problem-solving trajectories and their important segments for training AI models.
- Training on just 10% of the highest-quality data selected by SWE-Prime outperforms training on the full dataset, improving performance by up to 24.2% on some benchmarks.
- The method evaluates both entire problem-solving attempts and smaller step groups within those attempts to remove redundant, ineffective, or risky actions.
- This targeted approach reduces noisy or misleading supervision that can come from blindly using all successful problem-solving examples.
Large language models have shown promise in helping with software development tasks, such as debugging or writing code, by learning from examples of how problems were solved. Traditionally, researchers have gathered massive datasets of these “trajectories”—step-by-step records of how an AI or human agent solved a software problem—and then fine-tuned LLMs on all successful cases. However, not all successful trajectories are equally good teaching material. Some steps might be inefficient, redundant, or even risky, potentially teaching the model bad habits.
To address this, the SWE-Prime method introduces a two-stage process to carefully select training data. The first stage looks at entire trajectories and assesses them based on three criteria: process quality (how well the problem-solving steps were carried out), result quality (how good the final solution is), and data representativeness (how well the trajectory reflects common or important problem types). This filters out lower-quality or less relevant trajectories.
Next, the second stage breaks down the chosen trajectories into “semantic segments”—groups of consecutive steps that form meaningful sub-tasks or actions within the overall solution. Each segment is evaluated on its contribution to the final solution, how learnable it is (meaning how useful it is for the model to imitate), and whether it carries any risks (such as introducing errors). While all segments remain in the training sequence to keep context intact, only the selected high-quality segments are used to update the model’s learning during training. This nuanced approach allows the model to focus on the best parts of each trajectory without losing the bigger picture.
The researchers tested SWE-Prime on two software engineering benchmarks, SWE-Bench Pro and SWE-Bench Verified. Remarkably, using just 10% of the trajectories chosen by SWE-Prime led to better performance than using the entire dataset. Performance improvements reached up to 12.2% and 24.2% respectively, demonstrating the power of selective, quality-focused training data over sheer quantity.
This research highlights that more data isn’t always better for AI training—especially when that data includes suboptimal examples. By refining how training examples are selected and used, SWE-Prime offers a promising path toward more efficient and effective AI models for software problem-solving. Going forward, this method could help developers build smarter coding assistants and automated debugging tools that learn from the best practices rather than all practices. Further research might explore applying similar data selection techniques to other AI domains where quality and context matter.
Based on research published on arXiv by Dewu Zheng, Ruizhe Ye, Yanlin Wang et al..