Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
arXiv:1804.04235
Abstract
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.
References in corpus (4)
Cited by in corpus (66)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Scaling Laws for Neural Language Models
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Lookahead Optimizer: k steps forward, 1 step back
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- A Comparison of Optimization Algorithms for Deep Learning
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Assessing The Factual Accuracy of Generated Text
- WT5?! Training Text-to-Text Models to Explain their Predictions
- Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Talking-Heads Attention
- GSPMD: General and Scalable Parallelization for ML Computation Graphs
- Low-Memory Neural Network Training: A Technical Report
- Are Pre-trained Convolutions Better than Pre-trained Transformers?
- High Resolution Medical Image Analysis with Spatial Partitioning
- ETC: Encoding Long and Structured Inputs in Transformers
- An Adaptive and Momental Bound Method for Stochastic Learning
- Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trends
- AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
- Neural Language Modeling with Visual Features
- GLU Variants Improve Transformer
- Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
- TreeGen: A Tree-Based Transformer Architecture for Code Generation
- Weakly Supervised Grammatical Error Correction using Iterative Decoding
- Exploring Benefits of Transfer Learning in Neural Machine Translation
- AraGPT2: Pre-Trained Transformer for Arabic Language Generation
- Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
- Simple, Scalable Adaptation for Neural Machine Translation
- OmniNet: A unified architecture for multi-modal multi-task learning
- SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization
- Primer: Searching for Efficient Transformers for Language Modeling
- A Reinforced Generation of Adversarial Examples for Neural Machine Translation
- PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
- Contrastive Learning with Adversarial Perturbations for Conditional Text Generation
- Disentangling Adaptive Gradient Methods from Learning Rates
- Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models
- Investigating Multilingual NMT Representations at Scale
- HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
- Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
- Stochastic Optimization with Laggard Data Pipelines
- LocoProp: Enhancing BackProp via Local Loss Optimization
- DEUX: An Attribute-Guided Framework for Sociable Recommendation Dialog Systems
- Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
- Lecture Notes: Optimization for Machine Learning
- Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
- SAdam: A Variant of Adam for Strongly Convex Functions
- How Optimal is Greedy Decoding for Extractive Question Answering?
- Correcting diacritics and typos with a ByT5 transformer model
- CUNI System for the Building Educational Applications 2019 Shared Task: Grammatical Error Correction
- Evaluating Sequence-to-Sequence Learning Models for If-Then Program Synthesis
- Improving compute efficacy frontiers with SliceOut
- FH-SWF SG at GermEval 2021: Using Transformer-Based Language Models to Identify Toxic, Engaging, & Fact-Claiming Comments
- Corpora Generation for Grammatical Error Correction
- Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
- ur-iw-hnt at GermEval 2021: An Ensembling Strategy with Multiple BERT Models
- Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
- STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
- Domain-adversarial training of multi-speaker TTS
- Binary Search and First Order Gradient Based Method for Stochastic Optimization
- ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5
- Momentum Improves Normalized SGD
- XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
- FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learning