No Training Required: Exploring Random Encoders for Sentence Classification

Abstract

We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods---as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward---which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

Published as a conference paper at ICLR 2019

Cited by in corpus (32)

• Do Attention Heads in BERT Track Syntactic Dependencies?
• Pre-training via Paraphrasing
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Robust and Generalizable Visual Representation Learning via Random Convolutions
• Efficient and Private Federated Learning with Partially Trainable Networks
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Learning Compressed Sentence Representations for On-Device Text Processing
• On the Interpretability of Deep Learning Based Models for Knowledge Tracing
• Echo State Neural Machine Translation
• Contextual Lensing of Universal Sentence Representations
• Neural Language Priors
• On the impressive performance of randomly weighted encoders in summarization tasks
• Unsupervised Natural Question Answering with a Small Model
• CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
• Neural Machine Translation: A Review and Survey
• Sentence Embeddings by Ensemble Distillation
• Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English
• How Powerful Are Randomly Initialized Pointcloud Set Functions?
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• Text Classification and Clustering with Annealing Soft Nearest Neighbor Loss
• Cost-effective Deployment of BERT Models in Serverless Environment
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• A Revised Generative Evaluation of Visual Dialogue
• Deep Contextualized Self-training for Low Resource Dependency Parsing
• COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences
• Text Classification with Lexicon from PreAttention Mechanism
• Not all parameters are born equal: Attention is mostly what you need
• COSTRA 1.0: A Dataset of Complex Sentence Transformations