When to Fold'em: How to answer Unanswerable questions
arXiv:2105.00328
Abstract
We present 3 different question-answering models trained on the SQuAD2.0 dataset -- BIDAF, DocumentQA and ALBERT Retro-Reader -- demonstrating the improvement of language models in the past three years. Through our research in fine-tuning pre-trained models for question-answering, we developed a novel approach capable of achieving a 2% point improvement in SQuAD2.0 F1 in reduced training time. Our method of re-initializing select layers of a parameter-shared language model is simple yet empirically powerful.
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Bidirectional Attention Flow for Machine Comprehension
- Convolutional Neural Networks for Sentence Classification
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- A BERT Baseline for the Natural Questions