Translation, Sentiment and Voices: A Computational Model to Translate and Analyse Voices from Real-Time Video Calling

Abstract

With internet quickly becoming an easy access to many, voice calling over internet is slowly gaining momentum. Individuals has been engaging in video communication across the world in different languages. The decade saw the emergence of language translation using neural networks as well. With more data being generated in audio and visual forms, there has become a need and a challenge to analyse such information for many researchers from academia and industry. The availability of video chat corpora is limited as organizations protect user privacy and ensure data security. For this reason, an audio-visual communication system (VidALL) has been developed and audio-speeches were extracted. To understand human nature while answering a video call, an analysis was conducted where polarity and vocal intensity were considered as parameters. Simultaneously, a translation model using a neural approach was developed to translate English sentences to French. Simple RNN-based and Embedded-RNN based models were used for the translation model. BLEU score and target sentence comparators were used to check sentence correctness. Embedded-RNN showed an accuracy of 88.71 percentage and predicted correct sentences. A key finding suggest that polarity is a good estimator to understand human emotion.

arXiv comment79 Pages, 19 Tables, 24 Figures, A M.Sc Dissertation

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
• Addressing the Rare Word Problem in Neural Machine Translation
• A Comparison of Online Automatic Speech Recognition Systems and the Nonverbal Responses to Unintelligible Speech