Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization

Younes Bouhadjar, Maxime Fabre, Felix Schmidt +1

Linear recurrent neural networks have emerged as efficient alternatives to the original Transformer's softmax attention mechanism, thanks to their highly parallelizable training an…