The RNN Homunculus: Learning to Gate Long-Term Memory

Zeyu Fan, Alexandru-Dragos Manolache, Jan van Gemert

Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs a…