Learning to Generate Reviews and Discovering Sentiment
arXiv:1704.01444
Abstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Natural Language Processing (almost) from Scratch
- Convolutional Neural Networks for Sentence Classification
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Exploring the Limits of Language Modeling
- Visualizing and Understanding Recurrent Networks
- Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
- Object Detectors Emerge in Deep Scene CNNs
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
- Understanding Neural Networks through Representation Erasure
- Adversarial Training Methods for Semi-Supervised Text Classification
- Toward Controlled Generation of Text
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Towards Universal Paraphrastic Sentence Embeddings
- Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
- Inferring Networks of Substitutable and Complementary Products
- Multiplicative LSTM for sequence modelling
- Deep Learning with Dynamic Computation Graphs
Cited by in corpus (103)
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems
- DINOv2: Learning Robust Visual Features without Supervision
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
- Fine-Tuning Language Models from Human Preferences
- Understanding the Role of Individual Units in a Deep Neural Network
- From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
- An Analysis of Neural Language Modeling at Multiple Scales
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
- Generating and designing DNA with deep generative models
- LAMOL: LAnguage MOdeling for Lifelong Language Learning
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
- PrivFT: Private and Fast Text Classification with Homomorphic Encryption
- Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
- Multiplicative LSTM for sequence modelling
- Do Convolutional Networks need to be Deep for Text Classification ?
- Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
- Deep Learning-based Sentiment Classification: A Comparative Survey
- On Completeness-aware Concept-Based Explanations in Deep Neural Networks
- The emergence of number and syntax units in LSTM language models
- Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions
- Challenges in Building Intelligent Open-domain Dialog Systems
- Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding
- Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
- Towards falsifiable interpretability research
- Identifying and Controlling Important Neurons in Neural Machine Translation
- Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
- A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
- Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
- Combining Convolution and Recursive Neural Networks for Sentiment Analysis
- Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks
- Unsupervised Pretraining for Sequence to Sequence Learning
- A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning
- Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
- Learning to Generate Music With Sentiment
- Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations
- Distributional Generalization: A New Kind of Generalization
- Style Example-Guided Text Generation using Generative Adversarial Transformers
- Parallelizing Legendre Memory Unit Training
- What made you do this? Understanding black-box decisions with sufficient input subsets
- Senti-Attend: Image Captioning using Sentiment and Attention
- Breaking the Activation Function Bottleneck through Adaptive Parameterization
- On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations
- Explainable Software Defect Prediction: Are We There Yet?
- Machine Learning Techniques for Software Quality Assurance: A Survey
- Disentangled Representations for Manipulation of Sentiment in Text
- Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection
- Object category learning and retrieval with weak supervision
- Understanding Pre-trained BERT for Aspect-based Sentiment Analysis
- A journey in ESN and LSTM visualisations on a language task
- Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
- DONUT: CTC-based Query-by-Example Keyword Spotting
- Discovery of Natural Language Concepts in Individual Units of CNNs
- Cell-aware Stacked LSTMs for Modeling Sentences
- Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
- Surprisal-Triggered Conditional Computation with Neural Networks
- GeDi: Generative Discriminator Guided Sequence Generation
- Textual Membership Queries
- The geometry of integration in text classification RNNs
- Augmenting Neural Networks with First-order Logic
- FineText: Text Classification via Attention-based Language Model Fine-tuning
- Cooperative Learning of Disjoint Syntax and Semantics
- Learning Robust, Transferable Sentence Representations for Text Classification
- Improving Context Aware Language Models
- Unsupervised Natural Question Answering with a Small Model
- Low-Rank RNN Adaptation for Context-Aware Language Modeling
- Toward estimating personal well-being using voice
- Non-local Recurrent Neural Memory for Supervised Sequence Modeling
- A system for the 2019 Sentiment, Emotion and Cognitive State Task of DARPAs LORELEI project
- A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic Parsers
- Towards Controllable and Personalized Review Generation
- AspeRa: Aspect-based Rating Prediction Model
- Authorship Attribution in Bangla literature using Character-level CNN
- Learning to Embed Sentences Using Attentive Recursive Trees
- Assessing the Helpfulness of Review Content for Explaining Recommendations
- Music Classification in MIDI Format based on LSTM Mdel
- Using General Adversarial Networks for Marketing: A Case Study of Airbnb
- Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations
- How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
- Emergent Properties of Finetuned Language Representation Models
- Attribute Alignment: Controlling Text Generation from Pre-trained Language Models
- Towards Language Agnostic Universal Representations
- Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
- The Low-Dimensional Linear Geometry of Contextualized Word Representations
- Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
- Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?
- Affect in Tweets Using Experts Model
- Detecting Parkinson's Disease from interactions with a search engine: Is expert knowledge sufficient?
- Understanding the Importance of Single Directions via Representative Substitution
- Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
- AI-Powered Text Generation for Harmonious Human-Machine Interaction: Current State and Future Directions
- Automatically Exposing Problems with Neural Dialog Models
- Sparsity Emerges Naturally in Neural Language Models
- Semi-Supervised Self-Growing Generative Adversarial Networks for Image Recognition
- Towards Controlled Transformation of Sentiment in Sentences
- Meta-Learning a Dynamical Language Model
- Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
- Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
- Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
- Exploring Deep Neural Networks and Transfer Learning for Analyzing Emotions in Tweets
- Mining Program Properties From Neural Networks Trained on Source Code Embeddings
- SAM: Semantic Attribute Modulation for Language Modeling and Style Variation