Knowledge Distillation for Improved Accuracy in Spoken Question Answering
arXiv:2010.11067
Abstract
Spoken question answering (SQA) is a challenging task that requires the machine to fully understand the complex spoken documents. Automatic speech recognition (ASR) plays a significant role in the development of QA systems. However, the recent work shows that ASR systems generate highly noisy transcripts, which critically limit the capability of machine comprehension on the SQA task. To address the issue, we present a novel distillation framework. Specifically, we devise a training strategy to perform knowledge distillation (KD) from spoken documents and written counterparts. Our work makes a step towards distilling knowledge from the language model as a supervision signal to lead to better student accuracy by reducing the misalignment between automatic and manual transcriptions. Experiments demonstrate that our approach outperforms several state-of-the-art language models on the Spoken-SQuAD dataset.
To appear in ICASSP 2021
References in corpus (8)
- Distilling the Knowledge in a Neural Network
- Bidirectional Attention Flow for Machine Comprehension
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
- Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension
- Towards Data Distillation for End-to-end Spoken Conversational Question Answering
- Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension