Researchers have discovered a new way to improve how artificial intelligence (AI) agents learn to explore and solve tasks by changing what the AI focuses on during training. Traditionally, AI models trained with supervised fine-tuning pay attention only to the actions the agent takes—like pressing a button or moving forward—while ignoring the environment’s response to those actions. This new research shows that including the environment’s reactions as part of the training signals helps the AI better understand the consequences of its actions, leading to more effective learning when the AI later trains itself through trial and error (reinforcement learning).
Key Takeaways
- Standard training methods only supervise the AI’s chosen actions, not the environment’s responses, which may limit learning efficiency.
- The new approach, called ActObs, also trains the AI to predict environment observations, encouraging it to model cause and effect more accurately.
- ActObs leads to improved performance in downstream reinforcement learning tasks, including solving more problems and achieving higher success rates.
- This method helps the AI maintain a balanced understanding of its actions and their consequences, making learning more stable and effective.
In AI research, especially in reinforcement learning (RL), an agent interacts with an environment by taking actions and receiving observations or feedback in return. For example, a robot might move its arm (action) and then see the new position of an object (observation). Typically, when training these agents with supervised fine-tuning (SFT), the focus is only on matching the actions the agent took, while the environment’s observations are treated as background context rather than targets for prediction. This means the AI learns to imitate actions without explicitly learning to predict what will happen next in its surroundings.
The new method, ActObs, changes this by including the environment’s observations as part of what the AI must predict during supervised training. Although the AI never needs to generate these observations when deployed, training it to predict them encourages the policy—the AI’s decision-making process—to better model the consequences of its actions. This is done without adding extra data or making the AI model bigger or slower. Essentially, ActObs leverages information already present in the training data more fully, helping the AI build a richer internal understanding of the environment.
After this joint training, the researchers apply a reinforcement learning algorithm called GRPO to allow the AI to improve through exploration and feedback. They found that the AI initialized with ActObs outperforms the traditional action-only approach on several benchmarks, including code generation tasks and cross-domain code editing challenges that the AI had never seen before. Notably, ActObs-trained models retained more “entropy,” a term that here means the AI kept more diverse and flexible strategies during learning, which helped it explore better and avoid getting stuck in suboptimal behaviors. The researchers also observed that standard training causes the AI’s learning signals for actions and observations to become misaligned, while ActObs keeps them balanced, preserving the AI’s ability to predict the environment effectively.
This research suggests that integrating environment observations into supervised training could be a simple yet powerful way to improve AI agents’ learning and adaptability. By better understanding the consequences of their actions from the start, AI systems may become more efficient explorers and problem solvers in complex settings, from robotics to programming assistance. Future work could explore applying ActObs to other domains and larger models, potentially enhancing a wide range of AI applications that rely on interaction with dynamic environments.
Based on research published on arXiv by Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan et al..