93 citations · 314 across the 37 of their papers we have counts for
7 papers · 1 filter
Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
Wei Zhang, Mingrui Liu, Yu Feng +3
Distributed Deep Learning (DDL) is essential for large-scale Deep Learning (DL) training. Synchronous Stochastic Gradient Descent (SSGD) 1 is the de facto DDL optimization method.…
Improving Efficiency in Large-Scale Decentralized Distributed Training
Wei Zhang, Xiaodong Cui, Abdullah Kayi +9
Decentralized Parallel SGD (D-PSGD) and its asynchronous variant Asynchronous Parallel SGD (AD-PSGD) is a family of distributed learning algorithms that have been demonstrated to p…
Estimating Information Flow in Deep Neural Networks
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald +4
We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the informat…
A Comparison between Deep Neural Nets and Kernel Acoustic Models for Speech Recognition
Zhiyun Lu, Dong Guo, Alireza Bagheri Garakani +8
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition. Measuring perplexity and fra…
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
Zhiyun Lu, Avner May, Kuan Liu +8
The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems. We argue that this barrier can be effectively over…
Accelerating Hessian-free optimization for deep neural networks by implicit preconditioning and sampling
Tara N. Sainath, Lior Horesh, Brian Kingsbury +2
Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, bo…
Improvements to deep convolutional neural networks for LVCSR
Tara N. Sainath, Brian Kingsbury, Abdel-rahman Mohamed +6
Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also…