Artificial intelligence models, especially large language models, often excel in specific tasks but struggle to master multiple skills simultaneously. Researchers have been exploring ways to combine these specialized “expert” models into a single, versatile system. A newly published study dives deep into three popular methods for fusing these experts, offering fresh insights into how best to build multi-talented AI systems that balance performance and efficiency.
Key Takeaways
- Three main fusion methods—Merge, Mix RL, and Multi-Teacher On-Policy Distillation (MOPD)—each reuse different parts of expert models to consolidate capabilities across domains.
- While average performance differences among the methods are small, some tasks show notable gaps, highlighting the importance of choosing the right approach for specific needs.
- Merge is best for quick, cost-effective fusion when experts already exist; Mix RL suits training unified models from scratch with careful domain balancing; MOPD excels when preserving domain-specific expertise is a priority.
- All fusion methods improve accuracy on individual examples but do not expand the variety of solutions or reduce performance on tasks outside the training domains.
The challenge addressed by this research revolves around Reinforcement Learning with Verifiable Rewards (RLVR), a technique that enhances large language models by rewarding them for correct outputs in specific tasks. Although RLVR helps build strong domain experts, combining these experts into a single model that performs well across multiple areas remains complex. The study compares three fusion paradigms that differ in what parts of the experts they reuse:
- Merge: Combines the “task vectors,” which are specialized parameter updates representing each expert’s knowledge, into one consolidated model.
- Mix RL: Pools together the training data from different domains and trains a new model on this combined dataset.
- MOPD (Multi-Teacher On-Policy Distillation): Uses both the experts and their data by training a student model to imitate multiple expert teachers during reinforcement learning.
By evaluating these methods on a shared set of expert models and datasets across different model sizes and domains, the researchers uncovered nuanced trade-offs. For example, the Mix RL approach’s success depends heavily on how the training data from various domains is balanced, as uneven proportions can skew learning. The MOPD method’s performance is ultimately limited by the quality of its expert teachers—it cannot surpass them but can effectively preserve their strengths. Meanwhile, Merge efficiently compresses updates from all experts into a single vector, making it a cost-effective option when experts are already available.
Interestingly, all three methods improved the accuracy of predictions on individual examples but did not increase the overall range of problems the model could solve (solution coverage). Also, none of the fusion strategies caused the models to forget capabilities outside the domains they were trained on, suggesting they are safe to use without degrading general knowledge.
This research provides practical guidance for AI developers aiming to build multi-domain language models. If you already have strong domain experts and want a quick, inexpensive fusion, Merge is recommended. For creating a unified model from scratch, Mix RL allows more control over cross-domain learning by adjusting how much data comes from each domain. And when it’s crucial to maintain the best performance of each expert without exceeding training costs, MOPD is the preferred choice.
Looking ahead, these findings may influence how future AI systems are assembled, especially in applications requiring diverse skills like customer support, content creation, or scientific research. By understanding the strengths and limits of different fusion methods, developers can tailor AI models that are both versatile and reliable. Further research may explore how to extend these fusion techniques to even broader sets of domains or more complex tasks, pushing the boundaries of what multi-expert AI can achieve.
Based on research published on arXiv by Siye Wu, Kai Yang, Yuchen Cai et al..