QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

Abstract

Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q\&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.

arXiv commentPublished as full paper in ICLR 2018

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• Convolutional Sequence to Sequence Learning
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Highway Networks
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Dynamic Coattention Networks For Question Answering
• Layer Normalization
• Xception: Deep Learning with Depthwise Separable Convolutions
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Deep Networks with Stochastic Depth
• Depthwise Separable Convolutions for Neural Machine Translation
• Learning Recurrent Span Representations for Extractive Question Answering
• Multi-Perspective Context Matching for Machine Comprehension
• Making Neural QA as Simple as Possible but not Simpler
• Stochastic Answer Networks for Machine Reading Comprehension
• Ruminating Reader: Reasoning with Gated Multi-Hop Attention

Cited by in corpus (125)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Unsupervised Data Augmentation for Consistency Training
• Recent Trends in Deep Learning Based Natural Language Processing
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Passage Re-ranking with BERT
• The Natural Language Decathlon: Multitask Learning as Question Answering
• A Simple Method for Commonsense Reasoning
• Pathologies of Neural Models Make Interpretations Difficult
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• NSML: Meet the MLaaS platform with a real-world case study
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Quasi-hyperbolic momentum and Adam for deep learning
• Self-Attention Transducers for End-to-End Speech Recognition
• DRCD: a Chinese Machine Reading Comprehension Dataset
• ViGT: Proposal-free Video Grounding with Learnable Token in Transformer
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Multilingual Extractive Reading Comprehension by Runtime Machine Translation
• Entity-Relation Extraction as Multi-Turn Question Answering
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• Technical report on Conversational Question Answering
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
• Stochastic Answer Networks for Machine Reading Comprehension
• Dice Loss for Data-imbalanced NLP Tasks
• Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Machine Reading Comprehension: a Literature Review
• GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
• Neural Machine Reading Comprehension: Methods and Trends
• On Feature Normalization and Data Augmentation
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• Locality and compositionality in zero-shot learning
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Convolutional Self-Attention Networks
• Stochastic Answer Networks for SQuAD 2.0
• Incremental Transformer with Deliberation Decoder for Document Grounded Conversations
• Financial series prediction using Attention LSTM
• Question Answering over Knowledge Base using Language Model Embeddings
• SALSA-TEXT : self attentive latent space based adversarial text generation
• Generating Distractors for Reading Comprehension Questions from Real Examinations
• Improving Question Answering by Commonsense-Based Pre-Training
• Machine Reading Comprehension for Answer Re-Ranking in Customer Support Chatbots
• Attention-Guided Answer Distillation for Machine Reading Comprehension
• On the Generalization Effects of Linear Transformations in Data Augmentation
• Interactive Language Learning by Question Answering
• How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?
• Incorporating Structured Commonsense Knowledge in Story Completion
• FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Explicit Utilization of General Knowledge in Machine Reading Comprehension
• Improving Neural Question Generation using Answer Separation
• Learning to Impute: A General Framework for Semi-supervised Learning
• Lattice Transformer for Speech Translation
• Neural Sentence Ordering Based on Constraint Graphs
• RefNet: A Reference-aware Network for Background Based Conversation
• Cross-Lingual Transfer Learning for Question Answering
• QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
• Local and Global Context-Based Pairwise Models for Sentence Ordering
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification
• Making Neural Machine Reading Comprehension Faster
• BERTQA -- Attention on Steroids
• Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems
• Evaluating Rewards for Question Generation Models
• Text Embeddings for Retrieval From a Large Knowledge Base
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data
• FMA-ETA: Estimating Travel Time Entirely Based on FFN With Attention
• Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
• Contextual Aware Joint Probability Model Towards Question Answering System
• State of the Art in Fair ML: From Moral Philosophy and Legislation to Fair Classifiers
• Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hopping
• Multi-Instance Learning for End-to-End Knowledge Base Question Answering
• Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility
• GenNet : Reading Comprehension with Multiple Choice Questions using Generation and Selection model
• Unsupervised Domain Adaptation on Reading Comprehension
• Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification
• A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases
• DeepTag: inferring all-cause diagnoses from clinical notes in under-resourced medical domain
• Deep Human Answer Understanding for Natural Reverse QA
• LiveQA: A Question Answering Dataset over Sports Live
• Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
• Coarse to Fine: Video Retrieval before Moment Localization
• ConvDySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention and Convolutional Neural Networks
• BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels
• Knowledge Distillation for Improved Accuracy in Spoken Question Answering
• Toward Interpretability of Dual-Encoder Models for Dialogue Response Suggestions
• Learning Multi-level Dependencies for Robust Word Recognition
• Question Answering based Clinical Text Structuring Using Pre-trained Language Model
• A Study of the Tasks and Models in Machine Reading Comprehension
• Improving Span-based Question Answering Systems with Coarsely Labeled Data
• Attending to Emotional Narratives
• Weighted Global Normalization for Multiple Choice Reading Comprehension over Long Documents
• Interpretable Filter Learning Using Soft Self-attention For Raw Waveform Speech Recognition
• Weakly Supervised Domain Detection
• An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
• Mapping Natural Language Commands to Web Elements
• DCA: Diversified Co-Attention towards Informative Live Video Commenting
• Tag and Correct: Question aware Open Information Extraction with Two-stage Decoding
• Practical Annotation Strategies for Question Answering Datasets
• Why can't memory networks read effectively?
• Detecting ESG topics using domain-specific language models and data augmentation approaches
• ODSQA: Open-domain Spoken Question Answering Dataset
• Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning
• Hierarchical Semantic Correspondence Learning for Post-Discharge Patient Mortality Prediction
• QA4PRF: A Question Answering based Framework for Pseudo Relevance Feedback
• A New Sentence Ordering Method Using BERT Pretrained Model
• Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization
• NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
• Multiple Structural Priors Guided Self Attention Network for Language Understanding
• GrammarTagger: A Multilingual, Minimally-Supervised Grammar Profiler for Language Education
• Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase
• Incremental Reading for Question Answering