Researchers have developed a new approach to training language-based AI agents that could make them learn more effectively and efficiently. This is important because teaching AI to perform complex tasks over long periods typically requires a lot of trial and error, which is slow and costly. The new method, called CoPE, helps AI agents better use the information available during their learning process, potentially speeding up their ability to understand and act in diverse environments.
Key Takeaways
- CoPE introduces multi-step action targets that guide AI agents to predict sequences of future actions, not just immediate next steps.
- The method enables acting (making decisions) and forecasting (predicting future actions) to develop together within a single AI model.
- CoPE outperforms previous reinforcement learning techniques on five different benchmark tests, showing better overall success and faster learning.
- The approach achieves these improvements without needing extra models, separate pretraining phases, or added computational cost during use.
Traditional reinforcement learning (RL) methods for training language agents rely heavily on feedback given only at the end of a task—often a single number indicating success or failure. This makes it difficult for the AI to learn from the many steps taken along the way. Some recent methods try to provide more detailed guidance by estimating the value of each step or by predicting the environment’s responses, but these can be noisy or limited to short-term predictions.
The CoPE approach tackles this by introducing a new kind of training signal: multi-step action targets. Instead of just focusing on the immediate next action or the final outcome, CoPE trains the AI to generate sequences of future actions based on the history of what has happened so far. This means the AI learns to forecast not only what it should do next but also what it will likely do several steps ahead. By supervising these forecasts alongside actual actions, the AI’s decision-making model “co-evolves” with its ability to predict its own behavior.
Importantly, CoPE uses a shared policy backbone—a single core model—that learns from both acting and forecasting signals at the same time. During real use, the AI still makes decisions one step at a time, so there’s no extra computational burden. The researchers tested CoPE on five different simulated environments, including ALFWorld and ScienceWorld, which involve complex tasks like navigating virtual worlds or solving scientific problems. Across these tests, CoPE consistently outperformed baseline RL methods, sometimes doubling the speed at which the AI learned effective behaviors.
This research offers a promising direction for building smarter language agents that can learn faster and handle longer, more complex tasks without requiring expensive additional training resources. The CoPE method’s code is publicly available, encouraging further exploration and development. Looking ahead, this approach could enhance AI assistants, automated web agents, and other applications where understanding and planning multiple steps ahead is crucial. Future work might explore extending CoPE to even more diverse and realistic settings or combining it with other learning techniques to further improve AI performance.
Based on research published on arXiv by Haofei Yu, Yixing Chen, Yining Zhao et al..