Multi-modal Automated Speech Scoring using Attention Fusion

Abstract

In this study, we propose a novel multi-modal end-to-end neural approach for automated assessment of non-native English speakers' spontaneous speech using attention fusion. The pipeline employs Bi-directional Recurrent Convolutional Neural Networks and Bi-directional Long Short-Term Memory Neural Networks to encode acoustic and lexical cues from spectrograms and transcriptions, respectively. Attention fusion is performed on these learned predictive features to learn complex interactions between different modalities before final scoring. We compare our model with strong baselines and find combined attention to both lexical and acoustic cues significantly improves the overall performance of the system. Further, we present a qualitative and quantitative analysis of our model.

References in corpus (1)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Cited by in corpus (5)

• Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
• What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
• audino: A Modern Annotation Tool for Audio and Speech
• Towards Modelling Coherence in Spoken Discourse
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses