Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
arXiv:2109.00928 · doi:10.1145/3459637.3482395
Abstract
Automatic Speech Scoring (ASS) is the computer-assisted evaluation of a candidate's speaking proficiency in a language. ASS systems face many challenges like open grammar, variable pronunciations, and unstructured or semi-structured content. Recent deep learning approaches have shown some promise in this domain. However, most of these approaches focus on extracting features from a single audio, making them suffer from the lack of speaker-specific context required to model such a complex task. We propose a novel deep learning technique for non-native ASS, called speaker-conditioned hierarchical modeling. In our technique, we take advantage of the fact that oral proficiency tests rate multiple responses for a candidate. We extract context vectors from these responses and feed them as additional speaker-specific context to our network to score a particular response. We compare our technique with strong baselines and find that such modeling improves the model's average performance by 6.92% (maximum = 12.86%, minimum = 4.51%). We further show both quantitative and qualitative insights into the importance of this additional context in solving the problem of ASS.
arXiv commentPublished in CIKM 2021
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Axiomatic Attribution for Deep Networks
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Captum: A unified and generic model interpretability library for PyTorch
- Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
- Learning Private Neural Language Modeling with Attentive Aggregation
- Hierarchical Generative Modeling for Controllable Speech Synthesis
- SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring
- Personal VAD: Speaker-Conditioned Voice Activity Detection
- My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism
- What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
- Multi-modal Automated Speech Scoring using Attention Fusion
- Towards Modelling Coherence in Spoken Discourse
- Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency