Skip-Thought Vectors

Abstract

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

11 pages

References in corpus (7)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Distributed Representations of Sentences and Documents
• Exploiting Similarities among Languages for Machine Translation
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Self-Adaptive Hierarchical Sentence Model

Cited by in corpus (194)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Layer Normalization
• Deep Learning Based Text Classification: A Comprehensive Review
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
• RSVQA: Visual Question Answering for Remote Sensing Data
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• Adversarial Text-to-Image Synthesis: A Review
• Natural Language Processing Advancements By Deep Learning: A Survey
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
• The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
• Generalized Multi-view Embedding for Visual Recognition and Cross-modal Retrieval
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• How to Fine-Tune BERT for Text Classification?
• Finding beans in burgers: Deep semantic-visual embedding with localization
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Selfie: Self-supervised Pretraining for Image Embedding
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
• Correlating neural and symbolic representations of language
• Dialogue Act Classification with Context-Aware Self-Attention
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• DisSent: Sentence Representation Learning from Explicit Discourse Relations
• Enhancing Sentence Embedding with Generalized Pooling
• Look, Listen and Learn - A Multimodal LSTM for Speaker Identification
• Multimodal Entity Linking for Tweets
• Weakly-Supervised Multi-Level Attentional Reconstruction Network for Grounding Textual Queries in Videos
• Language learning using Speech to Image retrieval
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Uncovering Temporal Context for Video Question and Answering
• Contrastive Learning with Hard Negative Samples
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Multi-modality Latent Interaction Network for Visual Question Answering
• Multi-modal gated recurrent units for image description
• History Repeats Itself: Human Motion Prediction via Motion Attention
• Towards Making the Most of BERT in Neural Machine Translation
• NAIL: A General Interactive Fiction Agent
• Discovering the Compositional Structure of Vector Representations with Role Learning Networks
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• A Survey of Knowledge Enhanced Pre-trained Models
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Parameter-Efficient Transfer Learning with Diff Pruning
• Explaining Deep Neural Networks
• Sentence Centrality Revisited for Unsupervised Summarization
• Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• Residual Energy-Based Models for Text
• VirtualHome: Simulating Household Activities via Programs
• How to find a good image-text embedding for remote sensing visual question answering?
• Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent
• Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
• Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
• The Low-Rank Simplicity Bias in Deep Networks
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
• Learning Compressed Sentence Representations for On-Device Text Processing
• Image search using multilingual texts: a cross-modal learning approach between image and text
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism
• Extracting Symptoms and their Status from Clinical Conversations
• Learning to Infer Entities, Properties and their Relations from Clinical Conversations
• Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Context Aware Machine Learning
• Automatic Story Generation: Challenges and Attempts
• Correlation Coefficients and Semantic Textual Similarity
• Probing Neural Dialog Models for Conversational Understanding
• Pre-training Text Representations as Meta Learning
• Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
• Unsupervised Learning of Sentence Representations Using Sequence Consistency
• Deep Multimodal Feature Encoding for Video Ordering
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• Visual Question Answering on 360° Images
• A Cross-Domain Transferable Neural Coherence Model
• Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
• Text Embeddings for Retrieval From a Large Knowledge Base
• Fill in the Blanks: Imputing Missing Sentences for Larger-Context Neural Machine Translation
• Learning Dense Representations of Phrases at Scale
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
• Audio-Linguistic Embeddings for Spoken Sentences
• Contextual Lensing of Universal Sentence Representations
• UER: An Open-Source Toolkit for Pre-training Models
• Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
• Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks
• Wasserstein-Fisher-Rao Document Distance
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
• ntuer at SemEval-2019 Task 3: Emotion Classification with Word and Sentence Representations in RCNN
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• A Structured Distributional Model of Sentence Meaning and Processing
• On the impressive performance of randomly weighted encoders in summarization tasks
• SMRT Chatbots: Improving Non-Task-Oriented Dialog with Simulated Multiple Reference Training
• CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
• NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic
• Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks
• Discrete Argument Representation Learning for Interactive Argument Pair Identification
• Unsupervised Document Embedding via Contrastive Augmentation
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
• Structural-Aware Sentence Similarity with Recursive Optimal Transport
• Incorporating Visual Semantics into Sentence Representations within a Grounded Space
• IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation
• Data-driven models and computational tools for neurolinguistics: a language technology perspective
• On-Device Sentence Similarity for SMS Dataset
• Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
• Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation
• Sentence Embeddings by Ensemble Distillation
• Evaluating the Utility of Document Embedding Vector Difference for Relation Learning
• Automatic learner summary assessment for reading comprehension
• Global Textual Relation Embedding for Relational Understanding
• User-in-the-loop Adaptive Intent Detection for Instructable Digital Assistant
• Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue Task
• Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Efficient Sentence Embedding using Discrete Cosine Transform
• NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Sequence-to-Set Semantic Tagging: End-to-End Multi-label Prediction using Neural Attention for Complex Query Reformulation and Automated Text Categorization
• Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
• PAUSE: Positive and Annealed Unlabeled Sentence Embedding
• Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
• QuASE: Question-Answer Driven Sentence Encoding
• P-SIF: Document Embeddings Using Partition Averaging
• Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation
• Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
• Sensitive Information Detection: Recursive Neural Networks for Encoding Context
• An end-to-end Neural Network Framework for Text Clustering
• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings
• A Face-to-Face Neural Conversation Model
• Cross-Thought for Sentence Encoder Pre-training
• Stacked Convolutional Deep Encoding Network for Video-Text Retrieval
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• Learning Representations from Product Titles for Modeling Shopping Transactions
• Target-Oriented Deformation of Visual-Semantic Embedding Space
• AriEL: volume coding for sentence generation
• Learning Interpretable and Discrete Representations with Adversarial Training for Unsupervised Text Classification
• An Unsupervised Semantic Sentence Ranking Scheme for Text Documents
• Learning Probabilistic Sentence Representations from Paraphrases
• Learning Universal Representations from Word to Sentence
• Interpretable Structure-aware Document Encoders with Hierarchical Attention
• Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation
• Linking artificial and human neural representations of language
• Correlations between Word Vector Sets
• Parameter-free Sentence Embedding via Orthogonal Basis
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• Identifying Domain Adjacent Instances for Semantic Parsers
• Learning to Represent Bilingual Dictionaries
• Shareable Representations for Search Query Understanding
• Character 3-gram Mover's Distance: An Effective Method for Detecting Near-duplicate Japanese-language Recipes
• Efficient Sentence Embedding via Semantic Subspace Analysis
• Affect in Tweets Using Experts Model
• Document Similarity for Texts of Varying Lengths via Hidden Topics
• Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation
• A Study on Passage Re-ranking in Embedding based Unsupervised Semantic Search
• A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
• Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop
• Learning Outside the Box: Discourse-level Features Improve Metaphor Identification
• Prob2Vec: Mathematical Semantic Embedding for Problem Retrieval in Adaptive Tutoring
• Language Guided Networks for Cross-modal Moment Retrieval
• BURT: BERT-inspired Universal Representation from Twin Structure
• TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator
• ReadOnce Transformers: Reusable Representations of Text for Transformers
• Deep Robust Multilevel Semantic Cross-Modal Hashing
• PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders
• Semantic sentence similarity: size does not always matter
• Variation and generality in encoding of syntactic anomaly information in sentence embeddings
• Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models
• Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines
• A New Sentence Ordering Method Using BERT Pretrained Model
• A Temporal Variational Model for Story Generation
• Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification
• BURT: BERT-inspired Universal Representation from Learning Meaningful Segment
• Large-Context Conversational Representation Learning: Self-Supervised Learning for Conversational Documents
• A Local Similarity-Preserving Framework for Nonlinear Dimensionality Reduction with Neural Networks
• Multi-Granularity Representations of Dialog
• Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase
• How Sequence-to-Sequence Models Perceive Language Styles?