An Intelligent Web-Based Framework for Real-Time Indian Sign Language Recognition and Speech Translation with Interactive Learning and Performance Analytics

Abstract

The availability of accessible and interactive systems for sign language interpreting services between the deaf and hearing communities still poses a great challenge. To overcome this problem, in this paper, we are proposing a web-based framework which combines Indian Sign Language recognition, Text to Speech conversion, performance analytics and learning support in a single platform as SIGN2SPEECH. The proposed system uses the MediaPipe framework to extract the landmarks of the hands from both image, video, and real-time webcam feed, which allows for efficient gesture recognition with less complexity. For sign gesture classification, a light weight neural network is used and for the continuous recognition of sign gestures temporal filtering techniques are used to increase the stability of the prediction and decrease the number of errors in each frame. The developed framework allows multiple types of input such as upload images, upload videos, and interaction with the live webcam to communicate in sign language in real time. Through an integrated module that converts gestures into meaningful text and outputs into speech, they can be converted into meaningful text and corresponding speech outputs, thus to improve accessibility and communication effectiveness. The framework also includes a built-in analytics dashboard to provide visualization of model performance using confusion matrices, heatmaps, accuracy scores and other graphical tools, enhancing transparency and system evaluation. Moreover, all the acknowledged results and confidence scores are accumulated into a structured JSON-based history module, that can be used for future reference and analysis. An interactive learning platform with A–Z alphabet sign and 117 frequently used words from a custom sign language dataset is also proposed to be incorporated as another contribution of this work. The experimental results show that the system has a high recognition rate with high real-time response and usability. SIGN2SPEECH is a scalable and intuitive solution as it integrates the functions of recognition, translation, visualization and learning into a web-based interface. The framework has great potential to improve communication, education and social inclusion of people who are hard of hearing.