Attention-Guided Answer Distillation for Machine Reading Comprehension

Abstract

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models. Besides, existing approaches are also vulnerable to adversarial attacks. This paper tackles these problems by leveraging knowledge distillation, which aims to transfer knowledge from an ensemble model to a single model. We first demonstrate that vanilla knowledge distillation applied to answer span prediction is effective for reading comprehension systems. We then propose two novel approaches that not only penalize the prediction on confusing answers but also guide the training with alignment information distilled from the ensemble. Experiments show that our best student model has only a slight drop of 0.4% F1 on the SQuAD test set compared to the ensemble teacher, while running 12x faster during inference. It even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA benchmark.

arXiv commentTo appear at EMNLP 2018

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Teaching Machines to Read and Comprehend
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Machine Comprehension Using Match-LSTM and Answer Pointer
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Adversarial Examples for Evaluating Reading Comprehension Systems
• DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
• Data Distillation: Towards Omni-Supervised Learning
• Multi-range Reasoning for Machine Comprehension
• Robust Machine Comprehension Models via Adversarial Training

Cited by in corpus (9)

• Knowledge Distillation: A Survey
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Knowledge Squeezed Adversarial Network Compression
• Hint-Based Training for Non-Autoregressive Machine Translation
• Efficient Video Classification Using Fewer Frames
• A Study of the Tasks and Models in Machine Reading Comprehension