Improved Synthetic Training for Reading Comprehension
arXiv:2010.12776
Abstract
Automatically generated synthetic training examples have been shown to improve performance in machine reading comprehension (MRC). Compared to human annotated gold standard data, synthetic training data has unique properties, such as high availability at the possible expense of quality. In view of such differences, in this paper, we explore novel applications of synthetic examples to MRC. Our proposed pre-training and knowledge distillation strategies show significant improvements over existing methods. In a particularly surprising discovery, we observe that synthetic distillation often yields students that can outperform the teacher model.
11 pages, 2 figures
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Teaching Machines to Read and Comprehend
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Cost-Effective Active Learning for Deep Image Classification
- Deep Bayesian Active Learning with Image Data
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Learning Active Learning from Data
- Coresets for Scalable Bayesian Logistic Regression
- Selection via Proxy: Efficient Data Selection for Deep Learning
- Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation
- Pool-based sequential active learning with multi kernels