4 papers
Navigating Potholes with Geometry-Aware Sharpness Minimization
Simon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu +3
Sharpness-aware minimization (SAM) encourages flat minima by perturbing parameters along directions of high loss curvature, but treats all parameter directions uniformly, ignoring…
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu +2
Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners u…
Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Simon Dufort-Labbé, Pierluca D'Oro, Evgenii Nikishin +3
When training neural networks, dying neurons -- units becoming inactive or saturated -- are traditionally seen as harmful. This paper sheds new light on this phenomenon. By explori…
Myriad: a real-world testbed to bridge trajectory optimization and deep learning
Nikolaus H. R. Howe, Simon Dufort-Labbé, Nitarshan Rajkumar +1
We present Myriad, a testbed written in JAX for learning and planning in real-world continuous environments. The primary contributions of Myriad are threefold. First, Myriad provid…