Researchers have developed a new technique to help large language models (LLMs) explore a wider range of reasoning paths when solving complex problems. This advancement addresses a common challenge where these models, after being trained with reinforcement learning methods that use verifiable rewards, tend to become overconfident and narrow their focus, limiting their ability to find diverse and accurate solutions. By leveraging guidance from smaller, weaker language models, the new approach encourages broader exploration, potentially improving the performance of AI systems on tasks requiring deep reasoning.
Key Takeaways
- Large language models trained with reinforcement learning and verifiable rewards often suffer from “entropy collapse,” meaning they generate less diverse reasoning paths over time.
- The new method introduces partial reasoning steps from smaller, weaker models as prompts to the larger model, disrupting overconfidence and encouraging exploration of different solutions.
- Experiments on several mathematical problem benchmarks show that this approach consistently improves performance, especially when evaluating many possible answers (higher k values).
- This technique avoids the need for additional supervised fine-tuning, complex reward functions, or elaborate prompt engineering, making it simple and efficient.
Reinforcement Learning with Verifiable Rewards (RLVR) is a training approach that helps language models improve their reasoning abilities by rewarding them when their answers can be verified as correct. However, a side effect of RLVR is that models tend to reduce their “policy entropy,” a technical term referring to the variety or randomness in their answer generation. Lower entropy means the model becomes more predictable and sticks to a narrower set of reasoning paths, which can hurt its ability to find diverse correct answers.
To tackle this problem, the researchers propose a straightforward yet effective strategy: instead of letting the large model generate answers from scratch during training, they feed it partial reasoning sequences created by smaller, less capable models. These partial sequences, called “outer prefixes,” serve as unfamiliar starting points for the larger model. Because these prefixes come from a weaker model, they differ significantly from what the larger model would usually generate on its own, effectively shaking up its usual patterns and forcing it to explore new lines of reasoning.
This approach can be thought of as a form of guided exploration. In reinforcement learning, exploration is crucial for discovering better solutions, but it can be difficult to maintain when models become overly confident in their current strategies. By injecting these external prefixes, the method introduces controlled randomness and diversity without needing complicated adjustments to the training process.
The team conducted experiments on multiple math reasoning benchmarks, which are standard tests that involve solving complex problems step-by-step. Their results showed that models trained with this weak-model guidance consistently outperformed those trained with vanilla RLVR, especially when considering a larger number of candidate answers (referred to as pass@k metrics). This indicates that the method broadens the model’s reasoning coverage, allowing it to find more correct solutions across a wider range of possibilities.
Importantly, the new technique achieves these improvements without requiring additional supervised fine-tuning (SFT), complex reward engineering, or intricate prompt designs. This simplicity makes it appealing for practical applications where training resources or expertise may be limited.
Looking ahead, this research opens up new avenues for improving the reasoning capabilities of large language models by incorporating guidance from smaller models. Such hybrid training strategies could enhance AI systems used in education, scientific research, and automated problem-solving, where exploring diverse reasoning paths is key to success. As AI continues to evolve, methods that balance confidence with curiosity—like this one—may become essential tools for developing more reliable and versatile intelligent agents.
Based on research published on arXiv by Xingyu Shen, Huishuai Zhang, Peng Li et al..