Researchers have developed a new way to train advanced AI systems, called terminal agents, by gradually making their virtual training environments more challenging. This approach, known as environment evolution, helps these AI models learn better and tackle tougher problems over time. Since AI agents are only as good as the situations they practice in, creating increasingly difficult environments ensures they keep improving rather than plateauing. This breakthrough could lead to smarter, more capable AI that adapts to complex tasks more effectively.
Key Takeaways
- Traditional methods of training AI agents hit limits because environments become too easy as models improve, reducing learning opportunities.
- Environment evolution incrementally raises the difficulty of training scenarios off-policy, meaning it doesn’t rely solely on the agent’s current behavior.
- This method schedules the generation of harder environments in stages, providing continuous learning signals throughout training.
- Tests show significant performance gains—up to 18 percentage points—in AI models trained with environment evolution compared to previous techniques.
Training AI agents typically involves letting them interact with simulated environments where they can practice tasks and learn from successes and failures. However, as AI models get better, the same training environments no longer challenge them enough to promote growth. Existing approaches tried to address this by tweaking environments based directly on the agent’s current weaknesses during training, but this on-policy method has limitations—it can’t always keep up as the agent rapidly improves and may not generalize well to new situations.
To overcome these challenges, the research team introduced the concept of environment evolution. Instead of constantly adjusting environments based on the agent’s immediate performance, this approach evolves training scenarios in a planned, step-by-step manner, increasing difficulty off-policy—that is, independently of the agent’s current actions. This allows for a smoother progression of challenges that remain appropriately tough as the AI develops.
The researchers identified three key “evolution directions” that influence how difficult an environment is. These directions come from analyzing the AI’s multi-turn learning objectives—essentially, how it plans and executes a series of actions rather than just one step at a time. To implement this, they built a multi-agent system that engineers loops of interaction, systematically creating new, harder environments generation by generation during training.
They tested their method using several state-of-the-art AI models, including Hy4 preview, Claude Opus 5, and GPT-5.6 Sol. The evolved environments consistently proved more challenging than those produced by earlier techniques. When applied to Qwen3.6-27B and Qwen3.6-35B-A3B models, environment evolution led to substantial improvements—14.4 and 18.0 percentage points better performance on a benchmark called Terminal-Bench 2.1, which measures how well agents handle complex terminal tasks over long horizons.
This research, published on arXiv, offers a promising new direction for training smarter AI agents that can continuously learn and adapt. By evolving environments in a structured way, AI systems could better prepare for real-world applications requiring long-term planning and robust decision-making. Future work may explore applying environment evolution to a wider range of AI tasks and further refining the process to maximize learning efficiency.
Based on research published on arXiv by Zhiyuan Fan, Tinghao Yu, Yuanjun Cai et al..