Learning to Generate Reviews and Discovering Sentiment

Abstract

We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Natural Language Processing (almost) from Scratch
• Convolutional Neural Networks for Sentence Classification
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Object Detectors Emerge in Deep Scene CNNs
• Understanding Neural Networks through Representation Erasure
• TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
• Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
• Inferring Networks of Substitutable and Complementary Products
• Deep Learning with Dynamic Computation Graphs

Cited by in corpus (101)

• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• DINOv2: Learning Robust Visual Features without Supervision
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• Fine-Tuning Language Models from Human Preferences
• Understanding the Role of Individual Units in a Deep Neural Network
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• An Analysis of Neural Language Modeling at Multiple Scales
• Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
• Generating and designing DNA with deep generative models
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• PrivFT: Private and Fast Text Classification with Homomorphic Encryption
• Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
• Do Convolutional Networks need to be Deep for Text Classification ?
• Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
• Deep Learning-based Sentiment Classification: A Comparative Survey
• The emergence of number and syntax units in LSTM language models
• On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions
• Challenges in Building Intelligent Open-domain Dialog Systems
• Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding
• Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
• Towards falsifiable interpretability research
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Combining Convolution and Recursive Neural Networks for Sentiment Analysis
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks
• A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• Learning to Generate Music With Sentiment
• Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations
• Distributional Generalization: A New Kind of Generalization
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• Parallelizing Legendre Memory Unit Training
• What made you do this? Understanding black-box decisions with sufficient input subsets
• Senti-Attend: Image Captioning using Sentiment and Attention
• On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations
• Disentangled Representations for Manipulation of Sentiment in Text
• Explainable Software Defect Prediction: Are We There Yet?
• Machine Learning Techniques for Software Quality Assurance: A Survey
• Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Understanding Pre-trained BERT for Aspect-based Sentiment Analysis
• Object category learning and retrieval with weak supervision
• DONUT: CTC-based Query-by-Example Keyword Spotting
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• A journey in ESN and LSTM visualisations on a language task
• Discovery of Natural Language Concepts in Individual Units of CNNs
• GeDi: Generative Discriminator Guided Sequence Generation
• Cell-aware Stacked LSTMs for Modeling Sentences
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• Surprisal-Triggered Conditional Computation with Neural Networks
• Augmenting Neural Networks with First-order Logic
• Textual Membership Queries
• Cooperative Learning of Disjoint Syntax and Semantics
• The geometry of integration in text classification RNNs
• FineText: Text Classification via Attention-based Language Model Fine-tuning
• Unsupervised Natural Question Answering with a Small Model
• Low-Rank RNN Adaptation for Context-Aware Language Modeling
• Improving Context Aware Language Models
• Learning Robust, Transferable Sentence Representations for Text Classification
• Non-local Recurrent Neural Memory for Supervised Sequence Modeling
• Toward estimating personal well-being using voice
• A system for the 2019 Sentiment, Emotion and Cognitive State Task of DARPAs LORELEI project
• Music Classification in MIDI Format based on LSTM Mdel
• Assessing the Helpfulness of Review Content for Explaining Recommendations
• Authorship Attribution in Bangla literature using Character-level CNN
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Towards Controllable and Personalized Review Generation
• A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic Parsers
• Emergent Properties of Finetuned Language Representation Models
• Learning to Embed Sentences Using Attentive Recursive Trees
• AspeRa: Aspect-based Rating Prediction Model
• Using General Adversarial Networks for Marketing: A Case Study of Airbnb
• Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations
• Sparsity Emerges Naturally in Neural Language Models
• Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
• Semi-Supervised Self-Growing Generative Adversarial Networks for Image Recognition
• Towards Language Agnostic Universal Representations
• Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?
• Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
• The Low-Dimensional Linear Geometry of Contextualized Word Representations
• Towards Controlled Transformation of Sentiment in Sentences
• Mining Program Properties From Neural Networks Trained on Source Code Embeddings
• Detecting Parkinson's Disease from interactions with a search engine: Is expert knowledge sufficient?
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• Exploring Deep Neural Networks and Transfer Learning for Analyzing Emotions in Tweets
• Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
• Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
• Meta-Learning a Dynamical Language Model
• Affect in Tweets Using Experts Model
• AI-Powered Text Generation for Harmonious Human-Machine Interaction: Current State and Future Directions
• Automatically Exposing Problems with Neural Dialog Models
• SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Understanding the Importance of Single Directions via Representative Substitution
• Attribute Alignment: Controlling Text Generation from Pre-trained Language Models