ReZero is All You Need: Fast Convergence at Large Depth

Abstract

Deep networks often suffer from vanishing or exploding gradients due to inefficient signal propagation, leading to long training times or convergence difficulties. Various architecture designs, sophisticated residual-style networks, and initialization schemes have been shown to improve deep signal propagation. Recently, Pennington et al. used free probability theory to show that dynamical isometry plays an integral role in efficient deep learning. We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches. Although much simpler than its predecessors, this gate enables training thousands of fully connected layers with fast convergence and better test performance for ResNets trained on CIFAR-10. We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster on enwiki8.

References in corpus (12)

• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• On the Number of Linear Regions of Deep Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• All you need is a good init
• Fixup Initialization: Residual Learning Without Normalization
• The Emergence of Spectral Universality in Deep Networks
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Learning Identity Mappings with Residual Gates

Cited by in corpus (40)

• A Survey on Visual Transformer
• High-Performance Large-Scale Image Recognition Without Normalization
• Latent Space Data Assimilation by using Deep Learning
• Evolving Normalization-Activation Layers
• Self-Supervised Bug Detection and Repair
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• Understanding the Difficulty of Training Transformers
• Deep Transformers with Latent Depth
• Set Functions for Time Series
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• On Optimal Transformer Depth for Low-Resource Language Translation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
• LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech
• Normalized Attention Without Probability Cage
• Hypercomplex Image-to-Image Translation
• ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• On the Transfer of Disentangled Representations in Realistic Settings
• Generating Images with Sparse Representations
• Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs
• Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models
• MedGPT: Medical Concept Prediction from Clinical Narratives
• On Expressivity and Trainability of Quadratic Networks
• 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• Hierarchical Graph-Convolutional Variational AutoEncoding for Generative Modelling of Human Motion
• AI Enlightens Wireless Communication: Analyses, Solutions and Opportunities on CSI Feedback
• Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search
• Improvement of a dedicated model for open domain persona-aware dialogue generation
• Predicting the Behavior of Dealers in Over-The-Counter Corporate Bond Markets