Deep Learning for India's Diverse Sign Languages: A Comprehensive Review from CNNs to Transformers and the Future of Sign Recognition Systems

Gurusiddappa Hugar; Ramesh M. Kagalkar

doi:10.57159/jcmm.5.3.26708

Authors

Gurusiddappa Hugar Computer Science and Engineering, AGMR College of Engineering and Technology, Varur, Affiliated to Visvesvaraya Technological University, Karnataka, India https://orcid.org/0000-0002-6350-6581
Ramesh M. Kagalkar Information Science and Engineering, Nagarjuna College of Engineering and Technology, Bengaluru, Affiliated to Visvesvaraya Technological University, Karnataka, India https://orcid.org/0000-0001-9985-1275

DOI:

https://doi.org/10.57159/jcmm.5.3.26708

Keywords:

Sign Language Recognition, Deep Learning, Transformers, Regional Languages, Systematic Review, Assistive Technology

Abstract

Background: Sign languages are the primary communication mode for Deaf and Hard of Hearing (DHH) communities, yet automated recognition research has concentrated on American and British Sign Languages, leaving Indian Sign Language (ISL) and its regional variants comparatively neglected despite an estimated 18 million users.
Methods: Following PRISMA 2020 guidelines, a systematic search of multiple databases (Scopus, IEEE Xplore, ACM Digital Library, SpringerLink, and Google Scholar) was conducted in October 2025 for studies published between January 2020 and October 2025. From 287 identified records, 47 studies were retained after title and abstract screening (Cohen's κ = 0.87) and full-text assessment against predefined inclusion criteria. Study quality was appraised using an adapted QUADAS-2 framework.
Results: Reported accuracy across individual studies rose with architectural evolution: CNN-based models 79–92%, CNN-LSTM hybrids 88–97%, pose-based methods 89–94%, and transformer-based approaches 91–98% under controlled conditions. The distribution of research was highly uneven across the nine regional sign languages, with Telugu Sign Language showing a roughly 16-fold per-capita deficit relative to Kannada. Persistent barriers included continuous-recognition performance gaps, signer dependence (5–18% accuracy drops), environmental sensitivity (8–15% drops), and severe data scarcity for regional variants.
Conclusions: The review proposes the ISLR-Bench standardization framework and a 2026–2030 research roadmap spanning self-supervised learning, cross-lingual transfer, edge optimization, and community-centered design, alongside accessibility targets for government digital platforms and DHH employment.

References

[1] Census of India, "Data on disabled persons," 2011.

[2] P. F. Whiting, A. W. Rutjes, M. E. Westwood, et al., "QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies," Ann. Intern. Med., vol. 155, no. 8, pp. 529–536, 2011.

[3] M. J. Raihan, M. I. Labib, A. A. J. Jim, et al., "Bengali-Sign: A machine learning-based Bengali sign language interpretation for deaf and non-verbal people," Sensors, vol. 24, no. 16, p. 5351, 2024.

[4] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, "OpenPose: Realtime multi-person 2D pose estimation using part affinity fields," IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 1, pp. 172–186, 2021.

[5] S. Giri and A. Patil, "Marathi sign language recognition using MediaPipe and deep learning algorithm." Res. Square, preprint, 2024.

[6] ISLRTC and MeitY, "Sign learn: Mobile dictionary app for Indian sign language," tech. rep., ISLRTC and MeitY, 2023.

[7] S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, 2010.

[8] S. T. Sivakavi and R. I. Minu, "Tamil Lang TSP: Tamil Lang transformer neural text to sign production," Int. Arab J. Inf. Technol., vol. 21, no. 6, pp. 1143–1154, 2024.

[9] D. Moher, A. Liberati, J. Tetzlaff, and D. G. Altman, "Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement," PLoS Med., vol. 6, no. 7, p. e1000097, 2009.

[10] World Health Organization, "Deafness and hearing loss." Fact Sheet, 2023.

[11] Ministry of Social Justice and Empowerment, "Rights of persons with disabilities act," 2016.

[12] SIL India, "Assessment of regional language varieties in ISL," sil report, SIL International, 2008.

[13] Z. C. Lipton, J. Berkowitz, and C. Elkan, "A critical review of recurrent neural networks for sequence learning," arXiv:1506.00019, 2015.

[14] P. P. Waghmare, "Deep learning approach for combined Indian sign language recognition and video generation model," Int. J. Intell. Syst. Appl. Eng., vol. 12, no. 4, pp. 3296–3302, 2024.

[15] C. Lugaresi, J. Tang, H. Nash, et al., "MediaPipe: A framework for building perception pipelines," arXiv:1906.08172, 2019.

[16] K. Goyal and G. Velmathi, "Indian sign language recognition using MediaPipe holistic," arXiv:2304.10256, 2023.

[17] C. Sujatha, M. Patil, U. Madannavar, N. Jain, A. Badni, R. Angadi, and S. Gangisetty, "Indian regional sign language recognition," in Proc. 14th Indian Conf. Comput. Vis., Graph. Image Process. (ICVGIP), pp. 1–7, 2024.

[18] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., "An image is worth 16×16 words: Transformers for image recognition at scale," arXiv:2010.11929, 2021.

[19] K. Amrutha, P. Prabu, and R. C. Poonia, "LiST: A lightweight framework for continuous Indian sign language translation," Information, vol. 14, no. 2, p. 79, 2023.

[20] U. Rastogi, R. P. Mahapatra, and S. Kumar, "Advanced gesture recognition in Indian sign language using a synergistic combination of YOLOv10 with Swin Transformer model," Sci. Rep., vol. 15, no. 1, p. 32894, 2025.

[21] G. Hugar and R. M. Kagalkar, "Hybrid dual-stream deep learning approach for real-time KSL in healthcare," J. Inf. Syst. Eng. Bus. Intell., vol. 11, no. 3, pp. 393–406, 2025.

[22] S. Gobhinath and S. Sophia, "Tamil sign language identification using the auto-metric graph neural network optimized with golden eagle optimization," IETE J. Res., vol. 71, no. 6, pp. 2010–2019, 2025.

[23] J. M. Joshi and D. U. Patel, "GIDSL: Indian-Gujarati isolated dynamic sign language recognition using deep learning," SN Comput. Sci., vol. 5, p. 527, 2024.

[24] G. Hugar, R. M. Kagalkar, and A. Das, "Comparative study of hybrid deep learning models for Kannada sign language recognition," Int. J. Comput. Intell. Syst., vol. 18, p. 191, 2025.

[25] A. L. Vengassery, E. R. Perera, E. Jobin, S. S. Pulickal, and S. Sadanandan, "Aksharavaani: A Malayalam sign language detector," Int. J. Sci. Res. Sci. Technol., vol. 12, no. 12, pp. 319–331, 2025.

[26] K. Mandke and P. Chandekar, "Deaf education in India," in Deaf Education Beyond the Western World, Oxford, U.K.: Oxford Univ. Press, 2019.

[27] B. Frenay and M. Verleysen, "Classification in the presence of label noise: A survey," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 5, pp. 845–869, 2014.

[28] N. H. Shudha, F. T. Johura, A. Singha, and K. P. Ghosh, "Building an effective CNN for Bangla sign language," in AIP Conf. Proc., vol. 3327, p. 020013, 2025.

[29] P. Nedungadi, G. Dileep, M. Geetha, and R. Raman, "Vision transformer-powered conversational agent for real-time Indian sign language e-governance accessibility," Sci. Rep., vol. 15, p. 33055, 2025.

[30] K. K. Podder, M. E. H. Chowdhury, A. M. Tahir, Z. B. Mahbub, A. Khandakar, M. S. Hossain, and M. A. Kadir, "Bangla sign language (BdSL) alphabets and numerals classification using a deep learning model," Sensors, vol. 22, no. 2, p. 574, 2022.

[31] A. A. Darekar et al., "Marathi sign language recognition for physically disabled people," Int. J. Adv. Res. Sci. Commun. Technol., vol. 2, no. 7, pp. 1–8, 2022.

[32] A. Sridhar, R. G. Ganesan, P. Kumar, and M. Khapra, "INCLUDE: A large scale dataset for Indian sign language recognition," in Proc. 28th ACM Int. Conf. Multimedia (MM '20), pp. 1366–1375, 2020.

[33] A. Joshi, A. Bhat, S. Pradeep, P. Gole, S. Gupta, S. Agarwal, and A. Modi, "CISLR: Corpus for Indian sign language recognition," in Proc. 2022 Conf. Empirical Methods Natural Lang. Process. (EMNLP), pp. 10357–10366, 2022.

[34] Indian Sign Language Research and Training Centre (ISLRTC), "Development of ISL dictionary (10,000 terms)," annual report, ISLRTC, New Delhi, India, 2021.

[35] D. U. Patel and J. M. Joshi, "Deep learning based static Indian-Gujarati sign language gesture recognition," SN Comput. Sci., vol. 3, p. 380, 2022.

[36] H. K. Vashisth, T. Tarafder, R. Aziz, M. Arora, and Alpana, "Hand gesture recognition in Indian sign language using deep learning," Eng. Proc., vol. 59, no. 1, p. 96, 2023.

[37] R. M. Badiger, R. Yakkundimath, G. Konnurmath, and P. M. Dhulavvagol, "Deep learning approaches for age-based gesture classification in South Indian sign language," Eng. Technol. Appl. Sci. Res., vol. 14, no. 2, pp. 13255–13260, 2024.

[38] Census of India, "Linguistic survey of Karnataka," 2011.

[39] C. Hemanth, M. Jyothi, L. Adarsh, H. M. Shashikala, and N. Thejaswini, "Kannada sign language recognition using hand gestures," Int. Res. J. Eng. Technol., vol. 11, no. 4, pp. 1–8, 2024.

[40] N. Ajmi, A. B. Raj, B. Basina, S. Varsha, and V. S. Kumar, "Malayalam sign language recognizer," Int. J. Eng. Res. Technol., vol. 11, no. 5, pp. 1–6, 2022.

[41] B. Shabina, S. Sony, and R. R. Rajeev, "An approach for Malayalam sentence to Indian sign language generation using synthetic animations," in AIP Conf. Proc., vol. 3134, p. 020002, 2024.

[42] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 2, pp. 318–327, 2020.

[43] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks," in Proc. 23rd Int. Conf. Mach. Learn. (ICML), (Pittsburgh, PA, USA), pp. 369–376, 2006.

[44] B. Kang, S. Xie, M. Rohrbach, et al., "Decoupling representation and classifier for long-tailed recognition," in Proc. 8th Int. Conf. Learn. Represent. (ICLR), 2020.

[45] D. Sarma and M. K. Bhuyan, "Methods, databases and recent advancement of vision-based hand gesture recognition for HCI systems: A review," SN Comput. Sci., vol. 2, p. 436, 2021.

[46] C. Shorten and T. M. Khoshgoftaar, "A survey on image data augmentation for deep learning," J. Big Data, vol. 6, p. 60, 2019.

[47] S. Bhinganiy, R. Gunjal, S. Game, S. Gore, and S. Gawali, "Real-time Marathi sign language translation into text and speech using hand gesture recognition," Int. Res. J. Adv. Eng. Manag., vol. 3, no. 5, pp. 1946–1953, 2025.