With the plethora of AI models available today, it can be difficult for you to discern what makes a successful model. Now, thanks to work by NCI’s Dr. Baris Turkbey of NCI’s Center for Cancer Research and other members of the Prostate Imaging Reporting and Data System (PI-RADS) Steering Committee, you have some concrete guidelines for developing, evaluating, and reporting on AI-assisted prostate imaging models.

PI-RADS—a system for scoring prostate scans—helps ensure that researchers follow the same standards when collecting data, and it gives clinicians clear guidance on when to seek further, more invasive, diagnostics and treatment for their patients.

This committee is looking to expand the PI-RADs standards to apply to AI. They outline their findings in a new article, along with tips for putting these rules into practice. In this case, they detail how to apply standards to a diagnostics AI model (specifically, a model to identify men at risk for prostate cancer who haven’t yet undergone a biopsy to confirm those results).

According to the authors, the ideal model should adhere to a uniform checklist. Some highlights from this checklist include:

• Data requirements. Data should be de-identified and meet PI-RADs standards (version 2.1) and include all the necessary patient- and lesion-level metadata.
• Metrics of success. Measurements should go beyond a biopsy result (the typical “ground truth”) and include cancer detection rates, sensitivity and specificity for images, negative predictive value (which helps clinicians determine if they should conduct additional diagnostics), and how well the model compares with a human in the same role. It’s also important to know if the community endorses the model, in addition to how well it performs.
• Stability. Even after deploying in clinical practice, the model still needs ongoing testing to ensure it maintains its robustness over time.
• Regulatory approval. Both the U.S. Food and Drug Administration and European Union regulate AI for diagnostic purposes. See the FDA’s digital health center for additional key guidance on AI regulation.

As noted by Dr. Turkbey, “This study is a first step in defining basic guidelines for AI. Our recommendations reflect today’s best practices and target a specific use case that focuses on diagnostics. It’s important to keep in mind that technology is advancing very quickly.”

He added, “Setting expected performance metrics will be important for moving this technology forward. Our hope is that future studies will incorporate these standard measurements and report the outcomes accordingly. This will go a long way toward developing AI models that bring true value to diagnostic practice. We also expect that other use cases will soon follow, which look beyond diagnostics, to include models on selecting and planning treatment, as well as prognosis.”