Curious about how AI can bring a more human perspective to assessing cancer? Then here’s a story you won’t want to miss. The main character is “SMuRF,” (which stands for “swin transformer-based multimodal and multi-region data fusion framework”)—a new approach for evaluating images for head and neck cancer.

But, unlike its small blue namesake, this SMuRF’s primary attribute is a swin-transformer block approach that you can apply to both whole slide pathology images and computed tomography (CT) scans. The model is able to shift attention from 2-dimensions for pathology to 3-dimensions for radiology, and back again, learning and seamlessly integrating data.

According to the authors, the secret to SMuRF’s success is that it behaves similarly to how a human would interpret clinical information. It looks first at the pathology slide to gather information from a small area, then uses the CT image to gain a broader perspective of the tumor and nearby tissue and lymph nodes. It essentially switches back and forth between these two scales of resolution, building information on the cancer and predicting how it might progress.

SMuRF is the culmination of a collection of NCI-funded studies (see citations below). As noted by senior author, Dr. Anant Madabhushi, of Emory University, “Our three studies offer a compelling narrative. Our work in pathology helped us create a virtual biomarker for thousands of head and neck cancers. And our radiology work led to a new way of examining not just the tumor, but surrounding tissue and lymph nodes as well. Each component helped to build our current model, which stands to revolutionize how we approach treatment management of head and neck cancers.”

He added, “Although our model is aimed at one particular type of cancer, we believe the multiscale and multi-region imaging data fusion scheme we used with SMuRF could easily apply to other multimodal learning and data integration questions in cancer research.”