Using multimodal deep learning, researchers at Brigham and Women’s Hospital have shown improved exploration, biomarker discovery, and enhanced outcome prediction across 14 cancer types. The pan-cancer analysis offers ways to link pathology slides and genomic profiles to better predict patient outcomes.

To better develop joint image-omic biomarkers that researchers can use for cancer prognosis, the researchers proposed a deep learning approach that uses whole slide images (WSIs) blended with molecular profile features to determine the risk of cancer death. The team got all diagnostic WSIs and the related molecular and clinical data from The Cancer Genome Atlas. The Cancer Genome Atlas data are publicly accessible through the NCI Genomic Data Commons data portal and are part of the NCI Cancer Research Data Commons. 

The multimodal deep learning algorithm would also explain how histopathology features, molecular features, and their interactions correlate with low and high-risk patients. In their study, “Pan-cancer integrative histology-genomic analysis via multimodal deep learning,” the researchers highlighted three key discoveries:

• Blending different types of data improved prognostic models for most of the cancer types studied.
• Using a multi-data approach offered more insight into cancer’s progression than a single data source alone.
• Interpreting the data through machine modeling helped clarify the pathological and genomic characteristics, which led to a more accurate prognosis.

The researchers also developed the Pathology-Omics Research Platform for Integrative Survival Estimation (PORPOISE). PORPOISE is a tool that uses model explanations of both WSIs and molecular profile data to help discover new prognostic biomarkers. 

Looking ahead, the researchers will focus on:

• developing more focused prognostic models by creating larger multimodal data sets for individual disease models.
• adapting models to large, independent multimodal test cohorts.
• using multimodal deep learning to predict the response and resistance to treatment.

Dr. Tiffany Chen, one of the study's authors, received the NCI Ruth L. Kirschstein National Service Award, a training grant, in 2021. Dr. Chen is a board-certified Harvard pathologist, currently completing her postdoctorate research fellowship at the Brigham and Women’s Hospital.