4 papers · 1 filter
Overcoming Rank Collapse in Feedback Alignment
Gauthier Boeshertz, Razvan Pascanu, Claudia Clopath
Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation. Intere…
Navigating Potholes with Geometry-Aware Sharpness Minimization
Simon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu +3
Sharpness-aware minimization (SAM) encourages flat minima by perturbing parameters along directions of high loss curvature, but treats all parameter directions uniformly, ignoring…
Revisiting Adam for Streaming Reinforcement Learning
Florin Gogianu, Adrian Catalin Lutu, Razvan Pascanu
Learning from a sequence of interactions, as soon as observations are perceived and acted upon, without explicitly storing them, holds the promise of simpler, more efficient and ad…
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu +2
Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners u…