Breaking the Activation Function Bottleneck through Adaptive Parameterization
arXiv:1805.08574
Abstract
Standard neural network architectures are non-linear only by virtue of a simple element-wise activation function, making them both brittle and excessively large. In this paper, we consider methods for making the feed-forward layer more flexible while preserving its basic structure. We develop simple drop-in replacements that learn to adapt their parameterization conditional on the input, thereby increasing statistical efficiency significantly. We present an adaptive LSTM that advances the state of the art for the Penn Treebank and WikiText-2 word-modeling tasks while using fewer parameters and converging in less than half as many iterations.
Published as a conference paper at NeurIPS (NIPS) 2018
References in corpus (19)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Neural Architecture Search with Reinforcement Learning
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- An Analysis of Deep Neural Network Models for Practical Applications
- Predicting Parameters in Deep Learning
- A Neural Representation of Sketch Drawings
- Self-Normalizing Neural Networks
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Learning to Generate Reviews and Discovering Sentiment
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- On Multiplicative Integration with Recurrent Neural Networks
- Convolution by Evolution: Differentiable Pattern Producing Networks