As artificial intelligence continues to advance, one important question remains: Can AI models perform well enough to assist in critical fields like medicine? A newly published study explores this by testing several open-source language models on the challenging Spanish MIR exam, the medical residency entrance test. The research dives deeper than just accuracy, examining how well these models know when they might be wrong, whether medical-specific training helps, and if they actually use medical images included in the exam. Understanding these factors is crucial for healthcare systems considering deploying AI tools safely and effectively.
Key Takeaways
- Four open language models with 8 to 27 billion parameters answered 1,552 questions from multiple years of the MIR exam, with the largest models ranking roughly between the 600th and 3,000th human candidates.
- Even the models’ least confident answers were correct more often than the baseline break-even rate of 25%, meaning abstaining from answering was not beneficial.
- Medical fine-tuning reduced the models’ overconfidence but also lowered overall accuracy and did not improve their ability to distinguish correct from incorrect answers based on confidence scores.
- Removing or swapping the exam’s medical images did not impact model accuracy, suggesting these models did not meaningfully use the visual information provided.
The researchers focused on open models, which are AI systems whose code and weights are publicly available and can run on a health system’s own hardware. This contrasts with proprietary models that rely on cloud services. The team evaluated how these models performed on the MIR exam, which tests medical knowledge and reasoning for Spanish medical graduates seeking specialty training.
Using the exam’s official scoring rules, the models answered all questions from eight editions of the test, covering multiple-choice formats and including some with clinical images. The study assessed not just raw correctness but also the models’ confidence in each answer. Confidence here means the model’s internal estimate of how likely its chosen answer is correct. Ideally, a model’s confidence should correlate with actual correctness—high confidence when right, low when wrong—so users can trust or question its answers appropriately.
Additionally, the researchers explored “medical fine-tuning,” a process where models are further trained on medical texts to improve domain-specific knowledge. They also analyzed whether the models’ correct answers were based on sound clinical reasoning or flawed logic, and if the provided images influenced performance.
The findings showed that while the largest models performed reasonably well—comparable to mid-range human candidates—their confidence scores were imperfect indicators of correctness. Even with fine-tuning, the models struggled to reliably identify when they were likely to be wrong. Interestingly, some correct answers rejected plausible distractors for clinically unsound reasons that the model’s confidence did not reflect. Moreover, the presence or absence of images did not change accuracy, indicating these language models do not effectively process visual data in this setting.
These results highlight both promise and limitations. On one hand, open language models demonstrate a solid baseline ability on a demanding medical exam, suggesting potential as educational or decision-support tools. On the other hand, their imperfect confidence calibration and lack of meaningful image use raise caution for clinical deployment without further improvements. Future work may focus on enhancing models’ uncertainty estimation, integrating multimodal understanding, and refining clinical reasoning to better support healthcare professionals.
As AI continues to evolve, studies like this provide valuable insights into what current open models can and cannot do in real-world medical contexts. Healthcare systems aiming to adopt AI must consider these nuances to ensure safe, trustworthy, and effective assistance for medical training and patient care.
Based on research published on arXiv by Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera et al..