Exploding and vanishing gradients in deep neural networks: the effect of residual connections

Vivek S Borkar

The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection i…