The estimated total population of Pakistan is approximately 225 million, out of which approximately 1 million people are hearing impaired. Such people find it difficult to fit in the society due to their hearing disability. They also fail to experience their surrounding through hearing, resulting in a barrier in the world of sound. This project aims to bridge this gap by recognizing the surrounding sounds and converting them into corresponding images to assist the hearing impaired persons in understanding the sounds.
The project consists of two components, namely hardware and software. The software component is sub-divided into two parts: offline and online. The offline part is about collecting audio signals of ten different audio categories such as cats, dogs, owls, crows, birds, glass breaking, siren, car horn, children playing and drilling. These audio signals are used to train the Artificial intelligence (AI)/ Machine Learning (ML) methods. The ML methods used are Decision Tree (DT), Stochastic Gradient Decent (SGD), Naive Bayes (NB), Random Forest (RF), K-Nearest Neighbors (KNN), Logistic Regression (LR). All these methods are trained on Mel-Frequency Cepstral Coefficients (MFCCs), which are extracted as features from the audio signals.
According to the experimental results of the offline part, KNN outperforms all other ML methods in terms of accuracy. It achieves an accuracy of 98.75%. Based on these results, the KNN method is only used in the online part where the KNN-trained model is burned on the Raspberry Pi 4. The Raspberry Pi 4 is connected with a microphone to collect the surrounding audio signals. There after the audio signals are pre-processed for noise removal, extraction of MFCC features and then KNN inference on the input audio signal to recognize the surrounding audio category and displays the corresponding image as virtual scene on the LCD screen connected to Raspberry Pi for the hearing impaired and deaf person. This project is a step towards facilitating the deaf people to know about what is happening around them and can help to avoid accidents.