As artificial intelligence systems become more widespread in businesses, accurately measuring how well they perform in real-world settings is increasingly important. A new research paper introduces a fresh approach to evaluating enterprise AI by focusing on the entire delivery process—called the “serving route”—rather than just the AI model’s internal version or identifier. This matters because what companies actually use is a combination of the AI’s core algorithm plus the way it’s deployed, accessed, and integrated, all of which affect how well it works in practice.
Key Takeaways
- Traditional AI benchmarks focus only on model identifiers, ignoring how the AI is served or accessed, which can cause inaccurate performance measurements.
- The new IBIB protocol measures AI capability by verifying the full serving route—the complete path from request to response—before scoring performance.
- Including reliability (how often the system fails or succeeds) in scoring changes conclusions about AI effectiveness compared to ignoring failures.
- Tests on eleven AI systems revealed that identical models can behave differently depending on serving route choices, highlighting the importance of measuring the full delivery system.
Most current AI evaluations look at a model’s “checkpoint” or version number, assuming this alone defines its capabilities. However, in enterprise settings, AI is rarely used directly from these checkpoints. Instead, it’s wrapped in software layers, accessed through different tools, and connected to various data sources—collectively called the serving route. This route impacts precision, speed, reliability, and even what outputs the AI produces. The researchers argue that ignoring these factors leads to “measurement error” when benchmarking AI performance.
To address this, the team developed the IBIB (pronounced “eye-bee-eye-bee”) protocol, which has three main parts. First, a “gold-blind capability-binding preflight” step checks that the serving route can actually fulfill the evaluation tasks before any real testing begins. This ensures the AI system is ready and able to handle the workload. Next, a “reliability-inclusive first-pass scoring rule” factors failures into the performance score, rather than excluding them, giving a more realistic picture of effectiveness. Lastly, an “adjudication” process reviews results without bias toward any specific score, helping maintain fairness and accuracy.
The researchers applied IBIB to eleven different enterprise AI systems across 128 locked tasks involving documents, spreadsheets, charts, tools, and databases. They found that even when using identical AI weights (internal model parameters), different serving routes caused significant differences in results—sometimes passing or failing key evaluation criteria unpredictably. This showed that the advertised model identifier alone did not reveal practical performance limits. Additionally, scoring that included failures changed the ranking and interpretation of AI capabilities compared to traditional methods that ignore reliability.
By releasing the IBIB algorithms, classification tables, and schemas publicly, the authors aim to provide a standardized way for companies and researchers to measure AI systems as they are actually used, not just as they are designed. This approach could help organizations make more informed decisions when selecting or updating AI tools, ensuring they get systems that perform reliably in their specific environments.
Looking ahead, this research highlights the need for evaluation protocols that reflect the complexity of deploying AI in real business workflows. Future work may expand IBIB to cover more types of tasks, integrate with continuous monitoring systems, or adapt to emerging AI architectures. While not a silver bullet, measuring AI by serving route rather than model identifier is a promising step toward more transparent and practical AI assessment in enterprises.
Based on research published on arXiv by Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan.