Researchers have uncovered a subtle but important factor that can improve how large language models (LLMs) learn through reinforcement learning—a method where AI improves by trial and error. Their new study shows that exactly where rounding happens during the AI’s number crunching can make the difference between a smooth training process and one that becomes unstable or even collapses. This insight helps explain why some training setups unexpectedly fail and offers a practical way to fix these problems without overhauling the entire system.
Key Takeaways
- Small differences in rounding numbers during training and inference can destabilize reinforcement learning for large language models.
- Aligning rounding locations in three key parts of the model—RoPE, SwiGLU, and residual RMSNorm—prevents training collapse on BF16 precision.
- Adjusting rounding in just the RoPE component alone can stabilize training, even if other minor differences remain.
- The approach does not require changing the inference engine, core matrix operations, or the learning algorithm itself.
Large language models like those used for chatbots and text generation rely on complex numerical computations involving billions of parameters. To speed up training and save memory, these models often use a lower-precision number format called BF16 (bfloat16), which represents numbers with fewer bits than standard 32-bit floats. However, BF16 can introduce small rounding errors that accumulate during training, especially when reinforcement learning is involved. Reinforcement learning updates the model based on feedback from its own outputs, so any mismatch between how numbers are rounded during training versus inference can cause the learning to become unstable or “off-policy,” meaning the model is learning from data that doesn’t match its current behavior.
In this new study, the authors focus on the “rounding location”—the exact points in the model’s internal calculations where numbers are rounded from higher precision (FP32) down to BF16. They identify three critical regions inside the Qwen-family language models where rounding alignment matters most: RoPE (Rotary Positional Embeddings), SwiGLU (a type of activation function), and residual RMSNorm (a normalization technique). By keeping intermediate calculations in higher precision until the final output is rounded, and ensuring the rounding happens in the same places during both training and inference, they manage to prevent the training from collapsing.
This approach differs from previous solutions that tried to align the entire software stack or change the learning algorithm itself. Instead, it targets just specific parts of the computation pipeline. The researchers tested their method across four versions of the Qwen model and found that aligning rounding in all three regions fully stabilized training. Surprisingly, aligning only the RoPE path also sufficed to keep training stable in their experiments, even though some minor numerical differences remained elsewhere.
In practical terms, this means AI developers can improve reinforcement learning stability by carefully controlling where rounding happens, without needing to redesign the inference engine or alter fundamental mathematical operations. This is valuable because it allows existing highly optimized hardware and software to be used as is, reducing the overhead and complexity of retraining large language models.
Looking ahead, this research opens up new avenues for making large-scale reinforcement learning more reliable and efficient. As LLMs continue to grow in size and capability, ensuring stable training processes will be critical for deploying them safely and effectively. Future work may explore whether these rounding alignment principles apply to other model architectures or training techniques, potentially broadening their impact across the AI field.
Based on research published on arXiv by Yijing Ke, Tianle Cai, Ruosong Wang.