Researchers have long been fascinated by how large language models (LLMs)—the AI systems behind chatbots and text generators—acquire knowledge during their extensive training on massive text datasets. A newly published study sheds light on an intriguing factor that helps these models learn better: auxiliary views, or alternative ways of expressing the same information. Understanding this could improve how future AI systems are trained and perform.
Key Takeaways
- Simply repeating the same information during training is necessary for LLMs to learn effectively, but rephrasing or paraphrasing helps more when training batches are small.
- Replacing some repeated text with auxiliary views—different formulations of the same content—improves learning outcomes, even for recalling factual knowledge.
- The benefit of auxiliary views does not depend on how strong or accurate the model generating those views is.
- Two types of knowledge—contextual (related to surrounding information) and foundational (core facts)—play distinct roles in helping models learn when they have prior knowledge gaps.
The research team, led by Joseph Lee and colleagues, designed controlled experiments to untangle how repetition and paraphrasing affect knowledge acquisition during pre-training. Pre-training is the initial phase where LLMs learn language patterns and facts from vast amounts of text before being fine-tuned for specific tasks. While repetition of information is known to aid learning, this study explored whether presenting the same knowledge in varied ways—auxiliary views—could provide extra benefits.
To do this, the researchers fixed the total number of tokens (words or pieces of words) that the model saw during training and experimented with how those tokens were allocated. Instead of just repeating the exact same documents, some tokens were used to present auxiliary views—alternative phrasings or reformulations of the original content. Surprisingly, this strategy led to better learning results, including improved factual recall, suggesting that diversity in how knowledge is represented is valuable.
The study also investigated whether the quality of the “teacher” model generating these auxiliary views mattered. They found that even when the teacher model was not particularly strong, the auxiliary views still helped the pre-training model learn more effectively. This points to the inherent usefulness of varied perspectives on the same knowledge rather than relying solely on the accuracy of the source.
Another important insight was the identification of two forms of knowledge that aid learning: contextual knowledge, which helps the model understand information in relation to its surroundings, and foundational knowledge, which consists of core facts and concepts. These forms of knowledge are especially helpful when the model has gaps in what it already knows, allowing it to fill in missing pieces more efficiently.
On a technical level, the authors explored how these effects appear within the model’s internal layers, noting biases and compression patterns that reflect the influence of auxiliary views. While the details are complex, this mechanistic understanding supports the idea that diverse representations of information help the model organize and retain knowledge better.
This research highlights the importance of data diversity during pre-training, offering a plausible explanation for why large, varied corpora lead to more capable language models. By incorporating auxiliary views—different ways of expressing the same knowledge—future AI training approaches could become more efficient and effective. This work opens the door to new strategies for designing training data and better understanding the inner workings of LLMs as they continue to evolve.
Based on research published on arXiv by Joseph Lee, Yidi Huang, Dokyoon Kim et al..