Improved memory in recurrent neural networks with sequential non-normal dynamics

A. Emin Orhan, Xaq Pitkow

Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of desig…