Researchers have uncovered intriguing new insights into how AI language models process and express information after additional training, known as post-training. The study shows that while these models’ internal understanding of language may remain stable, the way they translate that understanding into answers can shift significantly. This discovery matters because it challenges common assumptions about how AI models learn and respond, with implications for improving their reliability and interpretability.
Key Takeaways
- Post-training often reduces the sensitivity of AI models’ answers, meaning they become less accurate at distinguishing between human and AI-written texts.
- Despite stable internal representations, the models’ “readout” mechanisms—how they convert knowledge into answers—can change substantially after fine-tuning.
- In some cases, post-training repairs weaknesses in the models’ ability to judge the truth of negated statements, improving answer accuracy.
- A relatively small amount of targeted training data can influence the direction of these changes, highlighting the importance of fine-tuning strategies.
The study focused on six widely used base language models and their versions after undergoing post-training, a common practice where a model is further trained on new data or tasks after its initial development. The researchers examined how well the models could distinguish between texts written by humans and those generated by AI, using German-language samples. They compared two ways of assessing the models’ knowledge: “linear probes” and the models’ own direct answers.
Linear probes are tools that analyze the internal layers of a language model to see what information it contains. Think of them as diagnostic tests that read the model’s “brain” without relying on the model’s final output. In contrast, the model’s direct answers are the actual responses it generates when asked a question. The researchers used concepts from signal detection theory—a framework for measuring how well a system can distinguish between different signals—to evaluate performance.
Interestingly, while the linear probes and simpler classifiers based on word patterns could reliably separate human-written from AI-written texts, the models’ own answers often performed worse and sometimes even reversed the expected order. After post-training, the sensitivity of the models’ answers decreased on average, meaning the models became less effective at this classification task. This suggests that post-training altered the way the models translate their internal knowledge into answers, rather than changing the underlying knowledge itself.
Further analysis showed that this shift occurred mainly during supervised fine-tuning, a stage where the model is trained with labeled examples to improve specific tasks. The researchers pinpointed early layers of the models as key contributors to these changes and found that information processed after the input text (in positions following the text) played a crucial role in transferring the change between training stages.
To make their findings more reproducible, the team also tested the models on a publicly available task: determining the truth of negated statements (e.g., “The sky is not blue”). Here, post-training had a different effect—it improved the models’ answers, fixing issues present in the base versions. This repair was also linked to changes in how the models read out information from deeper layers, especially from positions after the statement itself.
The study highlights that even a small amount of carefully chosen training data—such as 2,000 labeled examples of authorship judgments—can significantly influence how a model’s answers align with its internal representations. This opens up possibilities for refining AI models to be more consistent and interpretable by focusing on how knowledge is “read out” rather than just what is represented internally.
Looking ahead, these findings suggest that developers and researchers should pay close attention not only to what AI models learn but also to how they express that knowledge. Understanding and controlling the readout process could lead to more reliable AI systems, especially in sensitive applications like detecting misinformation or verifying authorship. Future work may explore new fine-tuning techniques that preserve or enhance the alignment between a model’s internal understanding and its external answers.
Based on research published on arXiv by Jochen Laubrock.
