Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form

Abstract

In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for $t$ time steps the weight gradient can be expressed as a rank-$t$ matrix, while the weight Hessian is as a sum of $t^2$ Kronecker products of rank-$1$ and $W^{T}AW$ matrices, for some matrix $A$ and weight matrix $W$. Also, we show that for a mini-batch of size $r$, the weight update can be expressed as a rank-$rt$ matrix. Finally, we briefly comment on the eigenvalues of the Hessian matrix.

23 pages, 4 figures

References in corpus (6)

• Deep Learning in Neural Networks: An Overview
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Caffe: Convolutional Architecture for Fast Feature Embedding
• On the difficulty of training Recurrent Neural Networks
• Going Deeper with Convolutions
• cuDNN: Efficient Primitives for Deep Learning

Cited by in corpus (5)

• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Batch Normalization Preconditioning for Neural Network Training
• Parallel Complexity of Forward and Backward Propagation
• Modular Block-diagonal Curvature Approximations for Feedforward Architectures
• The Outer Product Structure of Neural Network Derivatives