A comparative study of AI and expert radiologist performance for technical recall assessment in screening mammography

In 2024, my work was accepted for scientific presentation at RSNA 2024!

Purpose

Safeguards are needed to protect autonomous artificial intelligence (AI) rule-out systems for breast cancer screening from exams with image quality (IQ) deficiencies that could impair the sensitivity of the models. This work introduces an AI system that assesses IQ in full-field digital mammography (FFDM) exams and evaluates it against expert radiologists.

Materials and Methods

An AI system was trained to identify screening FFDM exams that require a technical recall due to inadequate IQ. The system evaluates positioning and blur, with features based on MQSA criteria as well as learned features. This model was evaluated on 1,100 screening exams from 2 independent held-out U.S. datasets (Dataset 1: 835 exams, from 2012-2019; Dataset 2: 265 exams, from 2002-2009) that were labeled as requiring recall or not by 5 breast fellowship-trained radiologists with at least 15 years of experience (used for testing only).

The performance of the model was first evaluated based on a reference standard of the median assessment of the 5 readers. To compare the AI and individual readers, we employ an evaluation methodology similar to leave-one-out cross-validation. In this method, we left out one reader at a time to establish a consensus based on the assessments of the other readers. Then, we compared the excluded reader and the AI model to this consensus. We repeated this for each reader to ensure an unbiased comparison between the AI and readers.

Results

The percentage of exams recalled (inadequate IQ) by the 5 readers ranged from 0.7% to 13.7% (Mean: 6.4% Standard Deviation: 4.8%). Pairwise agreement between the readers was low (Cohen’s kappa: 0.06-0.36). Based on a reference standard established by all 5 readers, the AI model achieved a sensitivity of 66.7% [48.2%, 82.0%], a specificity of 96.3% [94.9%, 97.3%], and an area under the receiver operating characteristic curve (AUC) of 0.92 [0.88, 0.96]. The sensitivity of the model increased when more readers labeled a sample as inadequate (3/5 readers: 56.5%, 13/23; 4/5 readers: 83.3%, 5/6; 5/5 readers: 100.0%, 4/4). Based on the cross-validation approach, the system exhibited comparable performance to the mean of the readers in sensitivity (AI: 48.9% [45.5%, 52.3%]; Mean Rad.: 45.7% [18.3%, 73.1%]) and specificity (AI: 96.1% [95.6%, 96.5%]; Mean Rad.: 95.0% [89.5%, 100.0%]). The system achieved an average AUC of 0.86 [0.85, 0.87].

Conclusions

This study demonstrates that an AI model can achieve comparable performance to expert radiologists in assessing IQ for mammography, overcoming the inherent challenges in the task’s subjectivity.

Clinical Relevance/Application

AI systems that provide an automatic check on IQ can assist with improving image acquisition and also help safeguard cancer-detecting AI models from exams with poor IQ.