Modal interaction attention-based multimodal fusion network for early prediction of response to neoadjuvant chemotherapy in breast cancer

Neoadjuvant chemotherapy is a systemic therapy for breast cancer. Early prediction of efficacy can help patients who will not benefit from it to make timely adjustments to their treatment regimen and reduce toxic side effects. Although numerous deep learning-based image classification methods have been developed in recent years, they often fail to effectively explore inter-modal correlations, which does not align with the clinician’s process. We propose a modal interaction attention-based multi-modal fusion network, composed of an encoder for extracting multi-sequence and cross-modal features and a decoder for fusing multi-modal features. Experiments on 214 cases of data collected from clinics demonstrate that the classification method achieves an area under the curve (AUC) of 0.898, outperforming other state-of-the-art methods. From our experimental results, the addition of modal interaction attention-based network effectively tackle the challenge of multi-modal data fusion. The proposed method has the potential to offer early prediction of NAC effects and realize more optimal treatment plans for patients.