Part II: Effect of different evaluation methods to the application of a computer-aided prostate MRI detection/diagnosis (CADe/CADx) device on reader performance

Jeffrey H Maki; Nayana U Patel; Ethan J Ulrich; Jasser Dhaouadi; Randall W Jones

doi:10.1067/j.cpradiol.2024.04.003

Part II: Effect of different evaluation methods to the application of a computer-aided prostate MRI detection/diagnosis (CADe/CADx) device on reader performance

Curr Probl Diagn Radiol. 2024 Apr 21:S0363-0188(24)00073-2. doi: 10.1067/j.cpradiol.2024.04.003. Online ahead of print.

Authors

Jeffrey H Maki¹, Nayana U Patel², Ethan J Ulrich³, Jasser Dhaouadi³, Randall W Jones³

Affiliations

¹ Department of Radiology, University of Colorado Anschutz Medical Center, 12401 E 17th Ave (MS L954), Aurora, CO 80045, USA. Electronic address: jeffrey.maki@cuanschutz.edu.
² University of New Mexico Department of Radiology, Albuquerque, NM, USA.
³ BOT IMAGE, Inc., Omaha, NE, USA.

PMID: 38702282
DOI: 10.1067/j.cpradiol.2024.04.003

Abstract

Introduction: The construction and results of a multiple-reader multiple-case prostate MRI study are described and reported to illustrate recommendations for how to standardize artificial intelligence (AI) prostate studies per the review constituting Part I¹.

Methods: Our previously reported approach was applied to review and report an IRB approved, HIPAA compliant multiple-reader multiple-case clinical study of 150 bi-parametric prostate MRI studies across 9 readers, measuring physician performance both with and without the use of the recently FDA cleared CADe/CADx software ProstatID.

Results: Unassisted reader AUC values ranged from 0.418 - 0.759, with AI assisted AUC values ranging from 0.507 - 0.787. This represented a statistically significant AUC improvement of 0.045 (α = 0.05). A free-response ROC (FROC) analysis similarly demonstrated a statistically significant increase in θ from 0.405 to 0.453 (α = 0.05). The standalone performance of ProstatID performed across all prostate tissues demonstrated an AUC of 0.929, while the standalone lesion level performance of ProstatID at all biopsied locations achieved an AUC of 0.710.

Conclusion: This study applies and illustrates suggested reporting and standardization methods for prostate AI studies that will make it easier to understand, evaluate and compare between AI studies. Providing radiologists with the ProstatID CADe/CADx software significantly increased diagnostic performance as assessed by both ROC and free-response ROC metrics. Such algorithms have the potential to improve radiologist performance in the detection and localization of clinically significant prostate cancer.

Keywords: Artificial Intelligence; CADe; CADx; Prostate cancer; Prostate cancer detection.