Researchers have uncovered new insights into how a popular technique for adapting large AI models, called Low-Rank Adaptation (LoRA), behaves differently when used in two common training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Understanding these differences is important because LoRA is widely employed to efficiently customize AI systems for various tasks, from language understanding to decision-making. By revealing how the underlying mathematical changes evolve during training, this study helps improve how AI models are adapted, potentially leading to better performance and more reliable outcomes.
Key Takeaways
- LoRA updates in reinforcement learning focus their changes more narrowly (spectral concentration) across all learning rates, while in supervised fine-tuning, the changes spread out more at moderate learning rates.
- Reinforcement learning causes the directions in which LoRA updates rotate (singular subspaces) to shift much faster than in supervised fine-tuning, especially at low learning rates.
- A technique called group-mean baseline subtraction, used in reinforcement learning, is identified as the key factor driving these contrasting behaviors.
- Tailoring LoRA updates to the specific dynamics of each training method—flattening the update spectrum in supervised fine-tuning and encouraging rotation outside the current update span in reinforcement learning—leads to improved performance on complex mathematical reasoning tasks.
To explore how LoRA behaves differently in supervised fine-tuning versus reinforcement learning, the researchers analyzed the “singular spectrum” and “singular subspaces” of the updates it makes to AI models. Simply put, the singular spectrum refers to how the magnitude of changes is distributed across different directions in the model’s parameter space, while singular subspaces represent the directions themselves. By tracking these across training, the team could see whether the model’s updates were focused narrowly or spread out, and how quickly the directions of change shifted.
One key finding was that reinforcement learning consistently concentrates updates in fewer directions, regardless of the learning rate, and rotates these directions more rapidly at smaller learning rates compared to supervised fine-tuning. To understand why, the team examined the role of group-mean baseline subtraction—a method used in reinforcement learning to reduce noise in gradient estimates by subtracting the average reward baseline. This technique was shown to tilt the balance between two competing forces during training: coherent fitting (systematic improvement) and gradient noise (random fluctuations). Specifically, it pushes the model toward more focused (spectrally concentrated) updates and faster shifts in update directions, while also favoring lower learning rates.
Armed with these insights, the researchers proposed different strategies for applying LoRA depending on the training context. For supervised fine-tuning, they recommend flattening the spectrum of the full update, which means spreading out the adaptation effort across many directions to capture diverse features. In reinforcement learning, they suggest flattening only the part of the update outside the current LoRA increment span to encourage faster rotation of update directions, helping the model explore new adaptations effectively. Experiments on mathematical reasoning tasks with multiple AI models showed that these tailored approaches generally outperform other methods, highlighting the benefits of understanding and leveraging LoRA’s dynamics.
This research sheds light on the subtle but important differences in how AI models learn under supervised and reinforcement paradigms when using LoRA. By fine-tuning the adaptation process to the unique characteristics of each training method, developers can improve model efficiency and effectiveness, especially in complex reasoning tasks. Future work may explore how these findings extend to other types of AI tasks and architectures, potentially guiding the development of more adaptable and robust AI systems across diverse applications.
Based on research published on arXiv by Chenruo Liu, Zihan Wang, Yijun Dong et al..