Parallel training of DNNs with Natural Gradient and Parameter Averaging

Abstract

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.

Accepted as workshop contribution to ICLR 2015. 12 pages plus 16 pages of appendices, International Conference on Learning Representations (ICLR): Workshop track, 2015. [2 sets of minor fixes post-publication.]

References in corpus (1)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Cited by in corpus (48)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• On the Convergence of Local Descent Methods in Federated Learning
• A Field Guide to Federated Optimization
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Variational Federated Multi-Task Learning
• Preconditioned Stochastic Gradient Descent
• Variance Reduced Local SGD with Lower Communication Complexity
• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• A Kronecker-factored approximate Fisher matrix for convolution layers
• Bayesian Learning for Deep Neural Network Adaptation
• A Multi-Modal Feature Embedding Approach to Diagnose Alzheimer Disease from Spoken Language
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks
• End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Multi-task Recurrent Model for Speech and Speaker Recognition
• Deep Speaker Feature Learning for Text-independent Speaker Verification
• Fast Transformers with Clustered Attention
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• Pkwrap: a PyTorch Package for LF-MMI Training of Acoustic Models
• Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learning
• OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline
• CytonRL: an Efficient Reinforcement Learning Open-source Toolkit Implemented in C++
• Multi-task Recurrent Model for True Multilingual Speech Recognition
• Collaborative Deep Learning Across Multiple Data Centers
• Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition
• Deep Factorization for Speech Signal
• Phonetic Temporal Neural Model for Language Identification
• A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate
• Collaborative Learning for Language and Speaker Recognition
• Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity
• Asynchronous Decentralized Distributed Training of Acoustic Models
• Feature Whitening via Gradient Transformation for Improved Convergence
• PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR
• Security and Privacy Preserving Deep Learning
• Online Second Order Methods for Non-Convex Stochastic Optimizations
• Empirical Evaluation of Parallel Training Algorithms on Acoustic Modeling
• Reducing the Model Order of Deep Neural Networks Using Information Theory
• X-Vectors with Multi-Scale Aggregation for Speaker Diarization
• A Random Gossip BMUF Process for Neural Language Modeling
• AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
• The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication
• Phone-aware Neural Language Identification
• Trade-offs of Local SGD at Scale: An Empirical Study
• Slim-DP: A Light Communication Data Parallelism for DNN
• PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks
• Combining Natural Gradient with Hessian Free Methods for Sequence Training