Designing a Hybrid CNN-Transformer Architecture for Robust Medical Image Diagnosis
DOI:
https://doi.org/10.68012/shem.v2i2.313Keywords:
Deep Learning, Hybrid CNN-Transformer, Medical Image Diagnosis, Convolutional Neural Network, Self-AttentionAbstract
Medical image diagnosis increasingly relies on deep learning against a background where convolutional neural networks capture local texture well but miss long-range spatial relationships, while transformer models need large labeled datasets and heavy computation that clinical settings rarely have. This study pursues the objective of designing a hybrid CNN-Transformer architecture that fuses convolutional feature extraction with self-attention to raise diagnostic accuracy on X-rays, MRI scans, and histopathology slides without exceeding the computational budget of real-world deployment. The proposed method pairs a lightweight convolutional backbone, which extracts local texture and edge features, with a transformer encoder that models global relationships across image regions, merging the two branches through a cross-attention fusion module before a classification head outputs the diagnosis. Training the architecture on publicly available medical imaging datasets and evaluating it against standalone CNN and transformer baselines using accuracy, sensitivity, specificity, and F1-score, the result reveals that the hybrid architecture beats both baselines, raising accuracy and sensitivity on cases with subtle abnormalities that need local texture recognition and broader spatial reasoning together, while inference time stays within a practical range for clinical workflows. These findings lead to the conclusion that the hybrid CNN-Transformer design gives clinicians a computationally feasible path to more robust medical image diagnosis, and the paper releases the architecture and source code to support reproducibility and further research toward accessible, trustworthy AI-assisted healthcare.
Downloads
References
[1] O. N. Manzari, H. Ahmadabadi, H. Kashiani, S. B. Shokouhi, and A. Ayatollahi, “Medvit: a robust vision transformer for generalized medical image classification,” Computers in biology and medicine, vol. 157, p. 106791, 2023.
[2] S. Mavaddati, “A hybrid approach for brain tumor classification: Enhancing mri-based diagnosis with cnn-transformer synergy,” Journal of AI and Data Mining, vol. 14, no. 1, pp. 37–49, 2026.
[3] J. W. Kim, A. U. Khan, and I. Banerjee, “Systematic review of hybrid vision transformer architectures for radiological image analysis,” Journal of Imaging Informatics in Medicine, vol. 38, no. 5, pp. 3248–3262, 2025.
[4] J. S. Zraqou, “A cnn–vision transformer fusion architecture for robust medical image recognition and clinical diagnostic understanding,” in Advances in Vision Foundation Models, Recognition, and Practical Systems. IGI Global Scientific Publishing, 2027, pp. 265–296.
[5] N. NAGABHOOSHANAM, C. SURYANARAYANA, S. MURTHY, K. SUBRAHMANYAM, A. SINGH, K. NAGU, A. VANATHI, J. PHILIP, and A. RAJARAM, “Hybrid cnn-transformer framework for real-time medical image reconstruction and diagnostic support in healthcare system.” Journal of Environmental Protection & Ecology, vol. 26, no. 6, p. 2405, 2025.
[6] Y. Memon and F. Zeng, “Hybrid-mednet: a hybrid cnn-transformer network with multi-dimensional feature fusion for medical image segmentation,” Physics in Medicine & Biology, vol. 70, no. 19, p. 195016, 2025.
[7] V. Agarwal, M. Lohani, A. S. Bist, L. Rahardja, M. Hardini, and G. Mustika, “Deep cnn–real esrgan: An innovative framework for lung disease prediction,” in 2022 IEEE Creative Communication and Innovative Technology (ICCIT). IEEE, 2022, pp. 1–6.
[8] V. Agarwal, M. Lohani, and A. S. Bist, “A novel deep learning technique for medical image analysis using improved optimizer,” Health Informatics Journal, vol. 30, no. 2, p. 14604582241255584, 2024.
[9] N. J. Ali, “A deep hybrid cnn-transformer model for multi-modal tumour detection and segmentation across medical imaging applications,” Int. J. Intell. Eng. Syst, vol. 19, p. 138, 2026.
[10] D. L. L. De Oliveira, T. A. A. Tosta, L. A. Neves, and M. Z. Do Nascimento, “Hybrid cnn-transformer models in histopathology image analysis: A scoping review,” IEEE Access, vol. 13, pp. 212887–212919, 2025.
[11] R. K. Hudiono and S. Watini, “Remote medical applications of artificial intelligence,” International Transactions on Artificial Intelligence, vol. 1, no. 2, pp. 182–187, 2023.
[12] A. Sharma, M. R. Mani, D. Chinthala, R. Thangjam, A. Bhilare, T. Singh, A. S. Bist et al., “Computer vision-based medical image segmentation using hybrid cnn and transformer architectures,” International Journal of Artificial Intelligence and Machine Learning, vol. 6, no. 2s, pp. 445–458, 2026.
[13] U. Rahardja, Q. Aini, A. S. Bist, S. Maulana, and S. Millah, “Examining the interplay of technology readiness and behavioural intentions in health detection safe entry station,” JDM (Jurnal Dinamika Manajemen), vol. 15, no. 1, pp. 125–143, 2024.
[14] X. Guo, X. Lin, X. Yang, L. Yu, K.-T. Cheng, and Z. Yan, “Uctnet: Uncertainty-guided cnn-transformer hybrid networks for medical image segmentation,” Pattern Recognition, vol. 152, p. 110491, 2024.
[15] M. Al Emran, R. Mahbub, E. J. Barsha, R. Akter, B. U. Islam, and M. M. U. I. Arikh, “A stabilized hybrid cnn-transformer framework with channel attention for multi-class retinal disease classification,” in 2026 IEEE 2nd International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN). IEEE, 2026, pp. 1–6.
[16] Q. Zhang, R. Li, P. Tang, J. Zhang, G. Yang, Z. Lin, M. Zhang, and M. Chen, “A hybrid cnn-transformer deep learning model for differentiating benign and malignant breast tumors using multi-view ultrasound images,” Technology in Cancer Research & Treatment, vol. 25, p. 15330338261447344, 2026.
[17] E. Altarawneh and E. Alshdaifat, “Adversarial robustness of cnn, transformer, and hybrid architectures in medical imaging ai,” in International Conference on Microelectronics, Electromagnetics and Telecommunication. Springer, 2025, pp. 88–99.
[18] M. Annas, T. Handra, C. S. Bangun, U. Rahardja, and N. Septiani, “Reward and promotion: Sustainable value of post pandemic efforts in medical cold-supply chain,” Aptisi Transactions on Technopreneurship (ATT), vol. 6, no. 1, pp. 109–118, 2024.
[19] D. Ashoka et al., “Effivit: Hybrid cnn-transformer for retinal imaging,” Computers in Biology and Medicine, vol. 191, p. 110164, 2025.
[20] O. N. Manzari, J. M. Kaleybar, H. Saadat, and S. Maleki, “Befunet: A hybrid cnn-transformer architecture for precise medical image segmentation,” arXiv preprint arXiv:2402.08793, 2024.
[21] S. Singh, “Computer-aided diagnosis of thoracic diseases in chest x-rays using hybrid cnn-transformer architecture,” arXiv preprint arXiv:2404.11843, 2024.
[22] R. S. Ali, R. K. Hassoun, A. Al-Adhami, S. A. Jabber, and S. H. Hashem, “Proposal medical image protection system based on hybrid cnn-transformer model and chaotic maps,” IEEE Access, vol. 13, pp. 123793–123807, 2025.
[23] S. A. Thanekar, “Hybrid cnn–transformer architecture for multi-organ disease classification,” Natural Resources for Human Health, vol. 6, no. 3, pp. 391–401, 2026.
[24] U. Rahardja, P. A. Sunarya, Q. Aini, S. Millah, and S. Maulana, “Technopreneurship in healthcare: Evaluating user satisfaction and trust in ai-driven safe entry stations,” Aptisi Transactions on Technopreneurship (ATT), vol. 6, no. 3, pp. 404–417, 2024.
[25] S. Dlamini, E. Kurniyaningrum, K. Kustiyono, U. Rahardja, Q. Aini, and S. Mahfudz, “Machine learningbased big data analytics for personalized health insurance services,” Smart Human-centered Emerging Research in Machine Intelligence, vol. 2, no. 1, pp. 26–36, 2026.
[26] H. Tang, Y. Chen, T. Wang, Y. Zhou, L. Zhao, Q. Gao, M. Du, T. Tan, X. Zhang, and T. Tong, “Htc-net: A hybrid cnn-transformer framework for medical image segmentation,” Biomedical Signal Processing and Control, vol. 88, p. 105605, 2024.
[27] T. K. Andiani and O. Jayanagara, “Effect of workload, work stress, technical skills, self-efficacy, and social competence on medical personnel performance,” Aptisi Transactions on Technopreneurship (ATT), vol. 5, no. 2, pp. 118–127, 2023.
[28] A. S. Anita, T. Kuusk, G. Nicola, M. Hardini, and U. Rahardja, “Advancements in artificial intelligence and their contributions to sustainable development goals: A multidisciplinary review,” Smart Humancentered Emerging Research in Machine Intelligence, vol. 2, no. 1, pp. 37–47, 2026.
[29] H. Long, “Hybrid design of cnn and vision transformer: A review,” in Proceedings of the 2024 7th International Conference on Computer Information Science and Artificial Intelligence, 2024, pp. 121– 127.
[30] R. R. Sharma, A. Sungheetha, M. Tiwari, I. A. Pindoo, V. Ellappan, and G. Pradeep, “Comparative analysis of vision transformer and cnn architectures in medical image classification,” in International Conference on Sustainability Innovation in Computing and Engineering (ICSICE 2024). Atlantis Press, 2025, pp. 1343–1355.
[31] I. Sembiring, B. K. Aji, and T. I. Bayu, “Consortium blockchain framework for secure digital medical record innovation,” Aptisi Transactions on Technopreneurship (ATT), vol. 8, no. 1, pp. 138–151, 2026.
[32] C. Hu, N. Cao, H. Zhou, and B. Guo, “Medical image classification with a hybrid ssm model based on cnn and transformer,” Electronics, vol. 13, no. 15, p. 3094, 2024.
[33] S. Juneja, B. S. Bhati, G. G. Tejani, and S. J. Mousavirad, “Vitcnn: a robust hybrid cnn–vision transformer based deep learning framework for multi-disease diagnosis in women’s healthcare,” Frontiers in Oncology, vol. 16, p. 1746864, 2026.
[34] A. Alwiyah and W. Setyowati, “A comprehensive survey of machine learning applications in medical image analysis for artificial vision,” International Transactions on Artificial Intelligence, vol. 2, no. 1, pp. 90–98, 2023.
[35] T. Behera, C. Parida, B. Kishan, M. Kar, A. Dash, and S. Darshana, “Edge-optimized explainable ai for medical image diagnosis using hybrid cnn-transformer architecture,” in 2025 5th IEEE International Conference on Applied Electromagnetics, Signal Processing, & Communication (AESPC). IEEE, 2025, pp. 1–6.
[36] A. H. Arribathi, S. Maulana, F. I. Ningrum, and M. Hassan, “Utilization of big data analytics in strategic decision making in the education sector,” Smart Human-centered Emerging Research in Machine Intelligence, vol. 2, no. 2, pp. 60–70, 2026.
[37] K. N. Erin, “A hybrid cnn-transformer approach for breast cancer detection and segmentation from ultrasound images,” Accessed: Dec, vol. 12, 2025.
[38] U. Rahardja, Q. Aini, D. Manongga, I. Sembiring, and I. D. Girinzio, “Implementation of tensor flow in air quality monitoring based on artificial intelligence,” International Journal of Artificial Intelligence Research, vol. 6, no. 1, 2023.
[39] Z. Yu, F. Lee, and Q. Chen, “Hct-net: hybrid cnn-transformer model based on a neural architecture search network for medical image segmentation: Z. yu et al.” Applied Intelligence, vol. 53, no. 17, pp. 19990– 20006, 2023.
[40] S. Iqbal, T. M. Khan, S. S. Naqvi, A. Naveed, and E. Meijering, “Tbconvl-net: A hybrid deep learning architecture for robust medical image segmentation,” Pattern Recognition, vol. 158, p. 111028, 2025.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Amelia Evans, Agung Rizky, Marta Rodriguez, Ihsan Nuril Hikam (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
