Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks

Abstract

We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam or AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.

Preprint, under review

References in corpus (21)

• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• Mixed Precision Training
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Large Batch Training of Convolutional Networks
• Pointer Sentinel Mixture Models
• Improving Generalization Performance by Switching from Adam to SGD
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Towards a Human-like Open-Domain Chatbot
• Qualitatively characterizing neural network optimization problems
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• ImageNet Training in Minutes
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Distinguishing general relativity from Chern-Simons gravity using gravitational wave polarizations
• Normalized Direction-preserving Adam
• Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network

Cited by in corpus (31)

• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Applications and Techniques for Fast Machine Learning in Science
• Learning Efficient Representations for Keyword Spotting with Triplet Loss
• Jasper: An End-to-End Convolutional Neural Acoustic Model
• Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
• Cross-Language Transfer Learning, Continuous Learning, and Domain Adaptation for End-to-End Automatic Speech Recognition
• A Deep Learning-Based Fully Automated Pipeline for Regurgitant Mitral Valve Anatomy Analysis From 3D Echocardiography
• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Using a thousand optimization tasks to learn hyperparameter search strategies
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Joint gender and age estimation based on speech signals using x-vectors and transfer learning
• Disentangling Adaptive Gradient Methods from Learning Rates
• The Geometry of Sign Gradient Descent
• Automated Learning Rate Scheduler for Large-batch Training
• Improving Noise Robustness of an End-to-End Neural Model for Automatic Speech Recognition
• Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control
• DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation
• Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition
• Federated Whole Prostate Segmentation in MRI with Personalized Neural Architectures
• Multi-QuartzNet: Multi-Resolution Convolution for Speech Recognition with Multi-Layer Feature Fusion
• Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
• Coarse-To-Fine And Cross-Lingual ASR Transfer
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging
• Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
• CarneliNet: Neural Mixture Model for Automatic Speech Recognition
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research