Designing a Modular Transformer-Based Framework for Low-Resource Language Text Summarization
DOI:
https://doi.org/10.68012/shem.v2i2.271Keywords:
Modular Transformer Architecture, Low-Resource Language Processing, Text Summarization, Adapter-Based Fine-Tuning, Cross-Lingual Transfer LearningAbstract
Text summarization for low-resource languages remains constrained by limited training corpora and the computational demands of large transformer models. This paper presents a modular transformer-based framework designed to address both constraints through architectural decomposition rather than data expansion. The framework separates the summarization pipeline into three interchangeable modules: a lightweight multilingual encoder pretrained on cross-lingual embeddings, a compression-aware attention mechanism that reduces parameter overhead by sharing weights across decoder layers, and a fine-tuning adapter layer that allows rapid domain adaptation with minimal labeled data. Each module operates independently, allowing researchers to swap components based on the target language's available resources without retraining the entire model. The design prioritizes transfer learning from high-resource languages, using adapter-based fine-tuning to bridge the performance gap that typically affects low-resource settings. We detail the architectural specifications, including layer configurations, attention head allocation, and the adapter injection strategy, alongside a reference implementation in Python using standard deep learning libraries. Rather than reporting benchmark results from empirical data collection, this study focuses on the design rationale, computational efficiency projections, and scalability considerations for languages with sparse digital text resources. The discussion extends to the framework's potential impact: reduced infrastructure barriers for NLP research in underrepresented languages, faster deployment cycles for organizations working in multilingual contexts, and a contribution toward more equitable access to AI-driven language technology. The proposed architecture offers a practical blueprint for researchers and practitioners aiming to build summarization systems where labeled data and computational resources are scarce.
Downloads
References
[1] N. C. Kundur and M. Kavitha, “Conversations with context: A modular transformer framework for understanding and generating natural language,” Procedia Computer Science, vol. 283, pp. 4426–4438, 2026.
[2] G. Moro, L. Ragazzi, L. Valgimigli, G. Frisoni, C. Sartori, and G. Marfia, “Efficient memory-enhanced transformer for long-document summarization in low-resource regimes,” Sensors, vol. 23, no. 7, p. 3542, 2023.
[3] A. Muhammad, A. Adeen, D. D. Arrova, and K. Shahreen, “A systematic review and experimental evaluation of classical and transformer-based models for urdu abstractive text summarization,” Information, vol. 16, no. 9, p. 784, 2025.
[4] B. Rawat, J. Nathalie, D. Manongga, G. Fransiso, F. P. O. Moana, and P. A. Sunarya, “Machine learningbased risk assessment and default prediction in p2p lending platforms,” Smart Human-centered Emerging Research in Machine Intelligence, vol. 2, no. 1, pp. 48–59, 2026.
[5] M. Munaf, H. Afzal, K. Mahmood, and N. Iltaf, “Low resource summarization using pre-trained language models,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 23, no. 10, pp. 1–19, 2024.
[6] S. V. Gawali and A. Mohanpurkar, “Large language model based text summarization: Taxonomy, evaluation practices, and research challenges,” INTERNATIONAL JOURNAL OF ADVANCES IN SIGNAL AND IMAGE SCIENCES, pp. 266–280, 2026.
[7] S. Masih, M. Hassan, L. G. Fahad, and B. Hassan, “Transformer-based abstractive summarization of legal texts in low-resource languages,” Electronics, vol. 14, no. 12, p. 2320, 2025.
[8] X.-H. Le, T. T. Do, and H.-Q. Le, “Graph-enhanced sentence retrieval for multi-document summarization in low-resource languages,” in Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2026, pp. 5281–5284.
[9] A. D. Kulkarni, S. T. Nikam, N. Shaikh, P. Senthil, V. Sumalatha, and T. P. Priyanka, “Multilingual foundation models for low-resource language translation,” in 2025 5th Asian Conference on Innovation in Technology (ASIANCON). IEEE, 2025, pp. 1–6.
[10] Z. Revesai, B. Ndlovu, and K. Maguraushe, “Transformer-based abstractive text summarisation for realtime web applications: A browser-integrated system with rest api architecture,” Journal of Applied Informatics and Computing, vol. 10, no. 3, pp. 2254–2268, 2026.
[11] C. Pramartha, I. M. Y. Mahendra, G. P. W. Rajeg, and I. W. Arka, “The development of semantic dictionary prototype for the balinese language,” International Journal of Cyber and IT Service Management, vol. 3, no. 2, pp. 96–106, 2023.
[12] U. Rahardja, Q. Aini, D. Manongga, I. Sembiring, and I. D. Girinzio, “Implementation of tensor flow in air quality monitoring based on artificial intelligence,” International Journal of Artificial Intelligence Research, vol. 6, no. 1, 2023.
[13] U. Hambali, R. Y. Natsir, E. Akib, and P. N. Valeryevna, “Language assessment literacy in efl edupreneurship among lecturers at muhammadiyah university of makassar and samarkand institute,” Aptisi Transactions on Technopreneurship (ATT), vol. 6, no. 3, pp. 598–607, 2024.
[14] A. H. Arribathi, S. Maulana, F. I. Ningrum, and M. Hassan, “Utilization of big data analytics in strategic decision making in the education sector,” Smart Human-centered Emerging Research in Machine Intelligence, vol. 2, no. 2, pp. 60–70, 2026.
[15] H. Ye, “Multilinguality and inclusive language technologies for low-resource languages,” Ph.D. dissertation, lmu, 2025.
[16] M. R. Anwar and H. A. Ahyarudin, “Ai-powered arabic language education in the era of society 5.0,” IAIC Transactions on Sustainable Digital Innovation (ITSDI), vol. 5, no. 1, pp. 50–57, 2023.
[17] U. Rahardja, Q. Aini, A. S. Bist, S. Maulana, and S. Millah, “Examining the interplay of technology readiness and behavioural intentions in health detection safe entry station,” JDM (Jurnal Dinamika Manajemen), vol. 15, no. 1, pp. 125–143, 2024.
[18] T. Yu, L. Yuan, H. Huang, J. Wang, and Y. Cai, “Mixture-of-experts low-rank adaptation for multilingual code summarization,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 2248–2260.
[19] A. Banerjee and D. Banik, “A comprehensive survey on transformer-based machine translation: Identifying research gaps and solutions for large language models,” ACM Computing Surveys, vol. 58, no. 5, pp. 1–33, 2025.
[20] U. Rahardja, S.-C. Chen, Y.-C. Lin, T.-C. Tsai, Q. Aini, A. Khan, F. P. Oganda, E. R. Dewi, Y.-C. Cho, and C.-H. Hsu, “Evaluating the mediating mechanism of perceived trust and risk toward cryptocurrency: An empirical research,” SAGE Open, vol. 13, no. 4, p. 21582440231217854, 2023.
[21] L. Hemalatha and V. P. Arivanantham, “Abstractive text summarization of nptel lectures for microlearning using fine-tuned t5 and beam search,” Procedia Computer Science, vol. 283, pp. 4263–4273, 2026.
[22] J. Harshini, N. Choudhary et al., “Inclusivity and accessibility focused text summarization,” in 2025 International Conference on Data Science and Business Systems (ICDSBS). IEEE, 2025, pp. 1–6.
[23] S. Dutta, A. Dutta, and C. Dev, “Bridging the resource gap: A review of transformer-based paraphrasing for low-resource languages with a focus on assamese,” Asian Journals of Electrical Sciences, vol. 15, no. 1, pp. 1–9, 2026.
[24] R. Feng, X. Hong, M. Jobanputra, M. Warning, and V. Demberg, “Retrieval-augmented modular prompt tuning for low-resource data-to-text generation,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 14 053–14 062.
[25] A. Sultana, J. R. Nisha, N. Nahyan, T. E. F. Hridi, S. Akter, and M. J. Abedin, “A deep learning approach to abstractive bangla text summarization,” in 2025 International Conference on Quantum Photonics, Artificial Intelligence, and Networking (QPAIN). IEEE, 2025, pp. 1–6.
[26] N. Hussain and G. A. Pangilinan, “Robotics and automation with artificial intelligence: improving efficiency and quality,” Aptisi Transactions on Technopreneurship (ATT), vol. 5, no. 2, pp. 176–189, 2023.
[27] A. S. Anwar, U. Rahardja, A. G. Prawiyogi, N. P. L. Santoso, and S. Maulana, “ilearning model approach in creating blockchain based higher education trust,” Int. J. Artif. Intell. Res, vol. 6, no. 1, 2022.
[28] F. Kabir, S. A. Mitu, S. Sultana, B. Hossain, R. Islam, and K. R. Ahmed, “Legalsummnet: A transformerbased model for effective legal case summarization.” International Journal of Advanced Computer Science & Applications, vol. 16, no. 9, p. 14, 2025.
[29] S. Mushtaq and K. Veningston, “Biomedical text summarization with large language models: methodologies, challenges, and future directions,” International Journal of Data Science and Analytics, vol. 22, no. 1, p. 29, 2026.
[30] U. Rahardja, I. D. Hapsari, P. H. Putra, and A. N. Hidayanto, “Technological readiness and its impact on mobile payment usage: A case study of go-pay,” Cogent Engineering, vol. 10, no. 1, p. 2171566, 2023.
[31] M. M. Yusuf, A. E. Chahari, A. M. Iliyasu, and H. Mikailu, “Transformer based abstractive text summarization for hausa: A low resource nlp approach,” Journal of Scientific Development Research, 2026.
[32] A. Magooda, “Techniques to enhance abstractive summarization model training for low resource domains,” Ph.D. dissertation, University of Pittsburgh, 2022.
[33] J. Siswanto, H. Hendry, U. Rahardja, I. Sembiring, and E. A. Lisangan, “Predicting transjakarta passengers with lstm-bilstm deep learning models for smart transportpreneurship,” Aptisi Transactions on Technopreneurship (ATT), vol. 7, no. 1, pp. 84–96, 2025.
[34] S. Wang, X. Sun, X. Li, and Y. Zhang, “Transformer–based models for low–resource language processing in specialized technical domains,” International Journal on Artificial Intelligence Tools, vol. 35, no. 03, p. 2650009, 2026.
[35] B. Rawat, A. S. Bist, N. Mehra, M. F. Fazri, and Y. A. Terah, “Study of kumaon language for natural language processing in end-to-end conversation scenario,” IAIC Transactions on Sustainable Digital Innovation (ITSDI), vol. 3, no. 2, pp. 143–149, 2022.
[36] S. Wijono, U. Rahardja, H. D. Purnomo, N. Lutfiani, and N. A. Yusuf, “Leveraging machine learning models to enhance startup collaboration and drive technopreneurship,” Aptisi Transactions on Technopreneurship (ATT), vol. 6, no. 3, pp. 432–442, 2024.
[37] A. E. Chahari, M. M. Elias, and U. D. Degereji, “Comparative analysis of deep learning and transformer models for low resource hausa language text summarization,” Journal of Arts and Sociological Research, 2026.
[38] U. Rahardja, P. A. Sunarya, Q. Aini, S. Millah, and S. Maulana, “Technopreneurship in healthcare: Evaluating user satisfaction and trust in ai-driven safe entry stations,” Aptisi Transactions on Technopreneurship (ATT), vol. 6, no. 3, pp. 404–417, 2024.
[39] N. Lutfiani, N. Fauziyah, F. P. Oganda, R. Setyaningrum, E. A. Natalia et al., “The role of globalization in indonesian evolution influence on media digital literacy language ai,” International Transactions on Artificial Intelligence, vol. 3, no. 2, pp. 192–200, 2025.
[40] A. Roy, F. H. Sunaira, T. Orpa, F. S. Yussouf, I. T. Naheen, and R. Khan, “Domain-adaptive bengali text summarization: Harnessing transformer-based large language models in closed-set contexts,” PloS one, vol. 21, no. 10, p. e0357890, 2026.
[41] W. B. Demilie and A. Zia, “News classification in low-resource languages: Insights from transformer and baseline models,” Concurrency and Computation: Practice and Experience, vol. 38, no. 6, p. e70648, 2026.
[42] U. Rahardja, M. Budiarto, K. Lutfiyah, O. F. P. Wahyudi, I. K. H. Azz, N. Azizah, and D. Julianingsih, “Analysis of the effectiveness of visual language and narrative in conveying value propositions in pitching decks,” International Transactions on Artificial Intelligence, vol. 3, no. 2, pp. 161–170, 2025.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Shofiyul Millah, Rio Squire Sean, Kristina Vaher, Dwi Julianingsih
This work is licensed under a Creative Commons Attribution 4.0 International License.
