As artificial intelligence models become central to evaluating other AI systems—acting as “judges” that score outputs or decide which data gets used for training—researchers have assumed these judge models provide stable, consistent measurements over time. A newly published study challenges that assumption, revealing that identical requests sent to the same AI model at different times often produce inconsistent results. This finding matters because many AI benchmarks, competitions, and quality controls rely on these “black-box” judge models to be reliable and repeatable. If they’re not, it raises questions about how we evaluate AI progress and make decisions based on these automated assessments.
Key Takeaways
- Repeated requests to the same AI judge model within a short time frame agreed only moderately (Spearman correlation around 0.40), far below the desired 0.90 consistency threshold.
- Replaying byte-identical inputs the next day improved consistency somewhat (around 0.78) but still fell short of the near-perfect 0.99 agreement expected.
- Inconsistencies arise from subtle internal mapping biases, noise in candidate rankings smaller than the instrument’s own noise floor, and unpredictable ranking changes even with identical inputs.
- Attempts to fix the problem by changing providers, waiting longer, or using different sampling methods did not reliably improve consistency; self-hosting helped only under very quiet server conditions.
The researchers conducted two large-scale, preregistered audits involving nearly 53,000 requests to judge models operating as black-box systems—meaning their internal workings were not accessible or modifiable. They tested whether the same input sent to the same model name would yield the same ranking or score when repeated immediately, the next day, or across different providers hosting the model. To ensure scientific rigor, all thresholds and evaluation criteria were fixed before data collection began (a process called preregistration).
“Judge” models are AI systems that evaluate other AI outputs, often used to gate training data, score generated text, or rank models on leaderboards. The key assumption underlying their use is that the model acts as a stable measurement instrument: if you ask it the same question twice, it should give the same answer. However, this study found that assumption does not hold on shared serving infrastructure where many users access the same model endpoint.
The authors identified three main mechanisms causing instability. First, the way the model maps internal label scores to meaningful rankings introduces bias comparable in size to the actual signal being measured. Second, the differences between candidate outputs that the model ranks are often much smaller than the noise inherent in the measurement process itself. Third, even when inputs are exactly the same byte-for-byte, the model sometimes returns different rankings, and this noise compounds when the order of candidates is permuted.
To explore possible fixes, the team tested various strategies: waiting longer between repeated requests, switching to different cloud providers hosting the same model, and substituting alternative metrics or sampling methods. None consistently resolved the reliability problem. They also tried self-hosting the model on a controlled kernel, which improved stability—but only when the server was not under heavy load.
Based on their findings, the authors offer a “snapshot-identity ladder” framework to better understand measurement stability at three levels, alongside eight design guidelines and a reporting checklist aimed at improving transparency and reliability in AI evaluation. They also highlight that a small-scale pilot test—just 2% of the total call volume—would have revealed these measurement issues before any evaluation gates were locked in.
These insights have important implications for AI research and development. Since many organizations rely on judge models to make decisions about model quality, training data selection, or leaderboard rankings, unstable measurements could lead to unfair or misleading conclusions. The study underscores the need to carefully validate the reliability of AI evaluators before using them as fixed instruments. Future work may focus on developing more robust measurement protocols or alternative architectures that reduce noise and variability, especially in shared infrastructure settings.
Based on research published on arXiv by Haoyaun Zhu, Jie Zhang.