394 citations · 895 across the 23 of their papers we have counts for
Showing 2017Show all
3 papers · 1 filter
cs.LG2017★ 5 cited
Parallel Complexity of Forward and Backward Propagation
Maxim Naumov
We show that the forward and backward propagation can be formulated as a solution of lower and upper triangular systems of equations. For standard feedforward (FNNs) and recurrent…
cs.LG2017★ 102 cited
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
Aditya Devarakonda, Maxim Naumov, Michael Garland
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally c…
cs.LG2017★ 8 cited
Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
Maxim Naumov
In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation thr…