As AI language models like ChatGPT become more interactive and widely used, researchers are uncovering new challenges in how we evaluate their performance. A newly published study dives into a subtle but important issue: users don’t just passively test these models—they adapt their behavior based on the model’s responses. This back-and-forth interaction can change the kinds of questions users ask over time, which in turn affects how well the model appears to perform. Understanding this dynamic is crucial for companies and developers who rely on evaluations to improve AI systems and decide which models to deploy.
Key Takeaways
- Users interacting with language models adapt their prompts based on previous responses, influencing subsequent evaluations.
- There are two ways to compare models: on a fixed set of prompts (common-distribution effect) or on prompts generated through user interaction (induced-distribution effect).
- Standard offline benchmarks and live A/B testing measure different performance effects and can lead to different model rankings.
- User adaptivity can increase variability in evaluation results and complicate interpretation of experiments.
Traditionally, AI models are tested using offline benchmarks—predefined sets of prompts where the model’s responses are scored. But this approach assumes that the prompts are fixed and independent of the model’s behavior. In real-world use, however, people adjust how they interact with the model based on what it says. For example, if a model struggles with certain question types, users might stop asking those or try to phrase them differently. This means the distribution of prompts changes with the model, a concept researchers call “prompt endogeneity.”
The study introduces two distinct effects to clarify this: the common-distribution effect compares models on the same prompt set, reflecting traditional offline evaluation. The induced-distribution effect looks at each model’s performance on the prompt distribution that emerges from users interacting with it live. The authors show that these two ways of measuring performance do not always agree. A model that performs best on a fixed benchmark might not be the top choice when considering how users actually change their behavior in response to it.
To explore these effects, the researchers use statistical tools and a simplified linear model of user–model interaction. They demonstrate that user adaptivity can increase the variance—or noise—in performance estimates, making it harder to confidently judge which model is better. They also identify conditions under which the rankings from offline tests and live experiments will agree, providing a practical guideline for when offline evaluation might suffice and when live testing is necessary.
Beyond the technical findings, the paper highlights broader implications. Adaptive user behavior can lead to phenomena like “reward hacking,” where users find ways to game the system, or “novelty effects,” where performance appears temporarily boosted as users experiment with new prompts. Recognizing these challenges is essential for interpreting experimental results fairly and for designing evaluation methods that reflect real-world use.
Looking ahead, this research suggests that AI developers and evaluators should measure not only how well a model performs on tasks but also how it influences user behavior. By accounting for the interactive nature of these systems, future evaluations can provide more reliable guidance for deploying AI tools that truly meet user needs.
Based on research published on arXiv by Hannah Li.