As artificial intelligence (AI) becomes more integrated into everyday technology, ensuring the security of AI models is increasingly important. Researchers have long relied on static security scanners to detect unsafe or malicious content hidden in AI artifacts—essentially the software pieces that make AI systems work. However, traditional ways of measuring these scanners’ effectiveness often focus only on how accurate their security judgments are, ignoring how often these tools can actually provide a judgment in the first place. A newly published study sheds light on this gap by evaluating three popular AI security scanners on a detailed benchmark, revealing important differences in their coverage, reliability, and failure recovery.
Key Takeaways
- Among three scanners tested—ModelAudit, Fickling, and ModelScan—ModelAudit made definitive security decisions on 100% of tested AI model families, while ModelScan only did so for about half.
- When ModelScan did provide a judgment, it was perfectly accurate, achieving 100% precision and recall, but it often failed to complete analysis on many samples.
- Fickling did not find any unique true positives beyond what ModelAudit and ModelScan identified, suggesting some redundancy between tools.
- For cases where ModelScan failed to analyze malicious models, the other two scanners successfully detected threats, highlighting the importance of combined tool use.
The research team created a synthetic benchmark consisting of 170 AI artifacts based on popular formats like Pickle and PyTorch, which are commonly used to save and share AI models. These artifacts were grouped into 145 “families,” with 135 having known security labels indicating whether they were safe or malicious. Ten additional artifacts were intentionally malformed to test how scanners handle uncertain or unsupported inputs. This setup allowed the researchers to go beyond traditional metrics like F1 score—which measures accuracy by combining precision and recall—and instead evaluate how often scanners could provide any security judgment at all, how often they completed their analysis, and how they handled unsupported or non-security-related findings.
In simpler terms, the study distinguished between two important concepts: judgment accuracy—how correct a scanner’s decision is when it makes one—and judgment availability—how often the scanner actually returns a usable decision. This distinction matters because a tool that is perfectly accurate but rarely provides results might be less useful in practice than one that offers good coverage with slightly lower accuracy. The researchers also looked at whether multiple tools add value by detecting unique threats or simply overlap in their findings.
ModelAudit stood out for its ability to analyze every AI model family and provide a definitive security decision, making it the most comprehensive in terms of coverage. ModelScan, while extremely accurate when it did provide results, often failed to finish its analysis, limiting its usefulness on its own. Fickling, meanwhile, did not detect any additional unique threats that were missed by the other two, suggesting that using all three tools together might not always improve detection beyond a certain point.
These findings have practical implications for organizations relying on AI security scanners to vet models before deployment. The study suggests that evaluating tools solely on accuracy metrics like F1 score can be misleading if the tool frequently fails to produce a judgment. Instead, security teams should consider both how often scanners complete their analyses and how well they recover from failures or unsupported inputs. Combining tools may help cover gaps, but understanding where overlaps occur is key to optimizing resources.
Looking ahead, the researchers highlight the need for more nuanced evaluation frameworks that account for coverage and failure recovery in AI security scanning. As AI systems continue to evolve and grow more complex, ensuring robust and reliable security assessments will be crucial for preventing malicious use and protecting users. Future work might explore how to improve scanner robustness or develop new metrics that better capture real-world effectiveness beyond traditional accuracy scores.
Based on research published on arXiv by Qianlong Lan, Vinothini Pandurangan, Anuj Kaul et al..