Training advanced AI models often requires enormous computational resources, but a new study explores whether a technique called on-policy distillation (OPD) can deliver similar or better performance with less computing power. The research, published recently by Zeyu Leo Liu, Yang Li, Bo Pang, and colleagues, compares OPD against a popular reinforcement learning method known as Group Relative Policy Optimization (GRPO) across various tasks and model sizes. Understanding which training approach is more compute-efficient could help developers build smarter AI systems while reducing energy use and cost.
Key Takeaways
- OPD generally underperforms GRPO in accuracy per unit of training compute when using an external, separately trained teacher model.
- When the teacher model is taken from the student’s own GRPO training checkpoint, OPD matches accuracy on math tasks using 50 times fewer floating-point operations (FLOPs) than GRPO.
- This high compute efficiency of OPD with a matched teacher also holds for larger students, code generation tasks, and a different model family.
- Combining multiple domain-specific GRPO-trained experts into a single student via multi-teacher OPD can maintain specialist-level accuracy with substantially less total compute.
Training AI models often involves reinforcement learning, where a model improves by trial and error, guided by rewards. GRPO is one such method that optimizes policies relative to groups of behaviors to improve learning stability and performance. On-policy distillation, by contrast, involves training a “student” model to mimic a “teacher” model’s outputs on data generated by the student itself, using a dense supervision signal at every token (word or code element) it produces. This approach freezes the teacher’s parameters and focuses training on the student, potentially reducing the computational burden.
The researchers set out to rigorously compare these two approaches by measuring how many floating-point operations (FLOPs)—a standard measure of compute effort—are needed to reach certain accuracy levels. They tested different combinations of task difficulty, model sizes, domains (like math problems and code generation), and optimization settings. A key variable was the choice of teacher model used in OPD: either an externally trained teacher or the student’s own checkpoint from GRPO training.
When the teacher was an external model trained separately, OPD consistently required more compute to achieve the same accuracy as GRPO and often produced longer, less efficient responses. However, when the teacher was the student’s own checkpoint derived from GRPO training, OPD became dramatically more efficient. For example, on math tasks, OPD achieved the same accuracy as GRPO but with 50 times fewer FLOPs. This efficiency gain also extended to larger student models, code generation tasks, and a different family of AI models, suggesting broad applicability.
While on a single domain the GRPO checkpoint is already the optimal model, making distillation an additive compute cost, the study highlights a more practical scenario: different domains often require different expert models. To address this, the authors experimented with multi-teacher OPD, where a student model learns from multiple domain-specific GRPO experts. This approach consolidated expertise across domains into one model while requiring only a fraction of the compute compared to training each expert from scratch. For example, distilling two domain experts cost just 5.2% of the specialist training FLOPs, and training a combined model for four domains cost 8.0% of the specialists’ compute. Overall, this pipeline matched joint GRPO performance on a two-domain mix with 42% fewer total FLOPs.
This research offers valuable insights for AI practitioners aiming to balance model performance with computational cost. By carefully selecting the teacher model and leveraging multi-teacher distillation, it may be possible to train versatile AI systems more efficiently. While OPD alone does not always outperform reinforcement learning methods like GRPO, its combination with GRPO-trained teachers and multi-domain consolidation presents a promising avenue for future AI development. Further work could explore scaling these techniques to even larger models and more diverse tasks, potentially helping reduce the environmental impact and financial cost of AI training.
Based on research published on arXiv by Zeyu Leo Liu, Yang Li, Bo Pang et al..