Automated polyp segmentation—identifying potentially dangerous growths during colonoscopy using AI—holds promise for improving early detection of colorectal cancer. However, a major challenge is that these AI systems can sometimes produce inaccurate results without any obvious warning, since there is no immediate “ground truth” to verify their outputs during real-time procedures. A newly published research paper proposes a novel method to estimate the reliability of polyp segmentation models at the moment they make predictions, potentially helping clinicians trust and safely use AI assistance in colonoscopy.
Key Takeaways
- The study introduces Referee-Based Quality Estimation (RBQE), a framework that measures how much two independently trained AI models agree on polyp segmentation in the same image, without needing a reference annotation.
- Even using a referee model with the same architecture but different training initialization provides a strong reliability signal, with an ROC-AUC of 0.923, indicating good ability to detect unreliable predictions.
- Using a referee model with a different architecture (SegFormer-B0) improves reliability detection further, reaching an ROC-AUC of 0.960 and outperforming traditional test-time augmentation methods.
- RBQE can selectively filter out low-agreement predictions, improving the overall accuracy of retained segmentations, and requires only a single extra model pass during inference, making it practical for real-time use.
In colonoscopy AI, “segmentation” means the model highlights which pixels in an image correspond to polyps. However, since doctors cannot instantly verify every AI output with a confirmed label during the procedure, errors may go unnoticed, potentially causing missed diagnoses or false alarms. To address this, the researchers developed RBQE, which employs a “referee” AI model trained independently of the primary segmentation model. The idea is simple: if both models agree strongly on the polyp regions, the prediction is likely reliable; if they disagree, the result should be treated with caution.
To test their approach, the team used a large external dataset of 1,223 colonoscopy images collected from four public sources. They experimented with different referee models to understand what factors affect agreement-based reliability. One key design axis was “referee independence”—whether the referee model is trained separately and initialized differently from the primary model. Another was “architectural diversity”—whether the referee uses a different AI architecture than the primary model.
The researchers found that even when the referee model shared the same architecture as the primary model but was independently trained, it provided a useful signal to flag unreliable segmentations. This suggests that having a second opinion from a similar but independently trained model already helps detect potential errors. When they used a referee with a different architecture, specifically SegFormer-B0, the reliability signal improved further, significantly outperforming both the same-architecture referee and existing test-time augmentation methods, which try to improve predictions by modifying input images during inference.
One interesting observation was that cases where both models predicted no polyp (empty masks) were trivially easy to agree on, so the team also evaluated RBQE’s performance excluding these straightforward examples. Even then, RBQE maintained a strong ability to identify unreliable predictions, and the advantage of cross-model agreement became even clearer.
Importantly, RBQE is efficient, requiring just one additional pass of the referee model at inference time, which is feasible in real-time clinical settings. It also supports “selective prediction”—clinicians or automated systems can choose to trust only high-agreement segmentations, potentially improving patient safety.
This research opens the door to more trustworthy AI tools in colonoscopy by providing a practical and interpretable way to gauge prediction reliability without needing ground-truth labels at deployment. Future work may explore integrating RBQE into clinical workflows and testing its impact on diagnostic accuracy and outcomes. As AI continues to advance in medical imaging, methods like RBQE will be crucial to ensure these technologies are safe and effective when used in real-world healthcare environments.
Based on research published on arXiv by Siddharth Gupta, Jitin Singla.