Intelligent Indian Sign Language Translator with Real-Time Gesture Recognition and Deep Learning

Abstract: Communication barriers between hearing-impaired individuals and the general public represent one of the most persistent challenges in inclusive society design. Indian Sign Language (ISL) serves as the primary expressive modality for approximately 18 million deaf individuals across India, yet its comprehension remains negligible among the general population. This paper presents the design, development, and rigorous evaluation of an Intelligent Indian Sign Language Translator System (ISLTS) that harnesses deep learning and computer vision to recognize hand gestures and translate them into text and synthesized speech in real time. The system employs a Convolutional Neural Network (CNN) trained on 7,500 custom ISL images augmented to 22,500 samples, achieving an overall gesture recognition accuracy of 92.4% and a mean average precision (mAP) of 0.89 across all gesture classes. MediaPipe Hands is integrated for real- time 21-point landmark detection, feeding a CNN classifier that operates at 28 frames per second on standard laptop hardware with latency below 0.5 seconds per prediction. A text-to-speech (TTS) module converts recognized gestures to audible output, enabling bidirectional communication. Comparative evaluation demonstrates that the proposed system outperforms sensor-based and earlier vision-based methods by 18–22 percentage points in accuracy while eliminating the need for specialized hardware. The proposed system offers a scalable, cost-effective, and non-intrusive solution with strong potential for deployment in educational institutions, healthcare settings, and public

Keywords: Indian Sign Language; Deep Learning; Gesture Recognition; Computer Vision; Real-Time Translation; Accessibility; Convolutional Neural Networks; MediaPipe; Text-to-Speech