Brief technical note on linearizing recurrent neural networks (RNNs) before vs after the pointwise nonlinearity

Marino Pagan, Adrian Valente, Srdjan Ostojic +1

Linearization of the dynamics of recurrent neural networks (RNNs) is often used to study their properties. The same RNN dynamics can be written in terms of the ``activations" (the…