Researchers have developed a new technique to combine multiple specialized language models into one versatile model without needing extra training, addressing a subtle but important issue called representation bias. This breakthrough could make it easier and faster to create AI systems that handle multiple tasks effectively, such as coding, math reasoning, and following instructions, by merging existing models more seamlessly.
Key Takeaways
- Combining multiple fine-tuned language models into a single multi-task model often leads to hidden representation bias, where the merged model’s internal states drift away from those of the original models.
- Previous fixes focused on encoder models (used in vision tasks), but decoder models (used in language generation) pose unique challenges due to their autoregressive nature and the importance of certain token positions.
- The new method, called Decoder-Aware Representation Tuning via Surgery (DARTS), uses a position-aware correction that emphasizes high-entropy (decision-critical) tokens, improving generation quality.
- Testing on code generation, math reasoning, and instruction-following tasks showed DARTS outperforms prior approaches while adding only 0.1% extra parameters, making it an efficient solution.
Modern large language models (LLMs) are often fine-tuned on specific tasks to improve performance—for example, one model might specialize in coding while another excels at math problem solving. Combining these specialized models into a single multi-task model can save resources and simplify deployment. However, this merging process can introduce “representation bias,” a mismatch in the internal hidden states of the merged model compared to the original individual models. This bias can degrade the quality of the generated text or code.
Previously, researchers tackled this problem for encoder-based models, which process input data all at once (common in vision tasks). However, decoder models—used in many language generation systems—generate text one token at a time, relying on previous outputs to predict the next word. This autoregressive nature creates new challenges. For one, errors can accumulate over the sequence of generated tokens, and not every token is equally important—some positions where the model must make critical decisions (high-entropy positions) have a bigger impact on the output quality.
To address these challenges, the authors of the newly published paper propose DARTS (Decoder-Aware Representation Tuning via Surgery). This method introduces a novel way to correct the representation bias by applying a position-dependent bias adjustment that respects the order of tokens and focuses more on the high-entropy positions—where the model’s decisions are less certain and more impactful. The correction uses an entropy-weighted L1 loss function, a mathematical way to measure and minimize the difference between the merged model’s hidden states and those of the source models, with greater emphasis on these critical positions.
The researchers tested DARTS on three challenging domains using Llama-2-7B models: HumanEval for code generation, GSM8K for mathematical reasoning, and AlpacaEval for instruction following. They found that DARTS significantly improved performance over existing surgery-based merging methods, without requiring additional training or a large increase in model size—the correction module added only 0.1% more parameters, making it a lightweight and practical solution.
This advancement could help AI developers more efficiently build multi-task language models by merging existing specialized models without extensive retraining, saving time and computational resources. It also highlights the importance of carefully handling the unique properties of decoder models when combining them. Future work may explore applying these ideas to even larger models or more diverse task combinations, potentially accelerating the development of versatile AI assistants capable of seamlessly switching between tasks.
Based on research published on arXiv by Aaryan Ajay Sharma, Sai Nishanth Padala, Seganrasan Subramanian.