Decoupling Gating from Linearity

Abstract

ReLU neural-networks have been in the focus of many recent theoretical works, trying to explain their empirical success. Nonetheless, there is still a gap between current theoretical results and empirical observations, even in the case of shallow (one hidden-layer) networks. For example, in the task of memorizing a random sample of size $m$ and dimension $d$, the best theoretical result requires the size of the network to be $\tilde{\Omega }(\frac{m^2}{d})$, while empirically a network of size slightly larger than $\frac{m}{d}$ is sufficient. To bridge this gap, we turn to study a simplified model for ReLU networks. We observe that a ReLU neuron is a product of a linear function with a gate (the latter determines whether the neuron is active or not), where both share a jointly trained weight vector. In this spirit, we introduce the Gated Linear Unit (GaLU), which simply decouples the linearity from the gating by assigning different vectors for each role. We show that GaLU networks allow us to get optimization and generalization results that are much stronger than those available for ReLU networks. Specifically, we show a memorization result for networks of size $\tilde{\Omega }(\frac{m}{d})$, and improved generalization bounds. Finally, we show that in some scenarios, GaLU networks behave similarly to ReLU networks, hence proving to be a good choice of a simplified model.

References in corpus (13)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Depth Creates No Bad Local Minima
• SGD Learns the Conjugate Kernel Class of the Network
• Optimistic Rates for Learning with a Smooth Loss
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• The Global Optimization Geometry of Shallow Linear Neural Networks

Cited by in corpus (8)

• Neural Networks Learning and Memorization with (almost) no Over-Parameterization
• Learning Parities with Neural Networks
• An Optimization and Generalization Analysis for Max-Pooling Networks
• On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models
• Towards Understanding Learning in Neural Networks with Linear Teachers
• Disentangling deep neural networks with rectified linear units using duality
• Deep Gated Networks: A framework to understand training and generalisation in deep learning
• Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning