Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Abstract

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

arXiv commentAccepted at ICLR 2018

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• A large annotated corpus for learning natural language inference
• Enriching Word Vectors with Subword Information
• Learning to Generate Reviews and Discovering Sentiment
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• What do Neural Machine Translation Models Learn about Morphology?
• Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Towards Universal Paraphrastic Sentence Embeddings
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• A Decomposable Attention Model for Natural Language Inference
• DisSent: Sentence Representation Learning from Explicit Discourse Relations
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Charagram: Embedding Words and Sentences via Character n-grams
• How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
• A Fast Unified Model for Parsing and Sentence Understanding
• Learning Generic Sentence Representations Using Convolutional Neural Networks
• Neural Paraphrase Identification of Questions with Noisy Pretraining
• Rethinking Skip-thought: A Neighborhood based Approach

Cited by in corpus (37)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XNLI: Evaluating Cross-lingual Sentence Representations
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Self-training Improves Pre-training for Natural Language Understanding
• Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
• Extracting Sentence Embeddings from Pretrained Transformer Models
• CORD19STS: COVID-19 Semantic Textual Similarity Dataset
• Semi-Supervised Sequence Modeling with Cross-View Training
• Named entity recognition architecture combining contextual and global features
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• A multi-task deep learning model for the classification of Age-related Macular Degeneration
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Multi-Task Learning in Histo-pathology for Widely Generalizable Model
• An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
• Task Selection Policies for Multitask Learning
• Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
• Improving Multilingual Semantic Textual Similarity with Shared Sentence Encoder for Low-resource Languages
• Joint Multi-Domain Learning for Automatic Short Answer Grading
• What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
• Learning Robust, Transferable Sentence Representations for Text Classification
• Evaluating Metrics for Bias in Word Embeddings
• Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain
• Improving Sentence Representations with Consensus Maximisation
• Knowledgeable Dialogue Reading Comprehension on Key Turns
• Learning Universal Sentence Representations with Mean-Max Attention Autoencoder
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Persistence Homology of TEDtalk: Do Sentence Embeddings Have a Topological Shape?
• Paraphrase Detection on Noisy Subtitles in Six Languages
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
• Adversarial Decomposition of Text Representation
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
• Fast Line Search for Multi-Task Learning
• Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models