Multi-Modal Transformer Attention for Precise Architectural Distortion and Prognostic Breast Cancer Prediction

This paper proposes a novel and fully explainable framework for the early prediction of breast cancer by detecting architectural distortion (AD) in mammograms. Our approach introduces a two-stage, Transformer-based system: first, a Swin-Unet module performs high-fidelity segmentation of AD in mammographic patches; second, a risk-stratification MLP fuses extracted image features with normalized clinical data (age, breast density) to estimate malignancy risk. To enhance clinical trust, the system delivers dual explanations: a precise segmentation overlay indicating “where” the distortion lies, and SHAP-based feature attributions revealing “why” the model assigned a particular risk score. Evaluated on the CBIS-DDSM and INbreast datasets, our framework is expected to outperform CNN-based baselines in both segmentation accuracy (Dice score) and classification performance (AUC), while maintaining interpretable decision-making aligned with clinical reasoning. These innovations—AD segmentation via Transformers, multimodal fusion, and transparent risk justification—represent a significant contribution to computer-aided diagnostics in breast oncology.