Parallel training of DNNs with Natural Gradient and Parameter Averaging

Abstract

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.

Accepted as workshop contribution to ICLR 2015. 12 pages plus 16 pages of appendices, International Conference on Learning Representations (ICLR): Workshop track, 2015. [2 sets of minor fixes post-publication.]

Cited by in corpus (16)

• On the Convergence of Local Descent Methods in Federated Learning
• Variance Reduced Local SGD with Lower Communication Complexity
• A Multi-Modal Feature Embedding Approach to Diagnose Alzheimer Disease from Spoken Language
• Deep Speaker Feature Learning for Text-independent Speaker Verification
• Fast Transformers with Clustered Attention
• OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline
• Multi-task Recurrent Model for True Multilingual Speech Recognition
• Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition
• Deep Factorization for Speech Signal
• Feature Whitening via Gradient Transformation for Improved Convergence
• Security and Privacy Preserving Deep Learning
• PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR
• AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
• Phone-aware Neural Language Identification
• PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks
• Slim-DP: A Light Communication Data Parallelism for DNN