2 citations · 4 across the 11 of their papers we have counts for
8 papers · 1 filter
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2
Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearit…
Toward a First-Principles Update Geometry for the Language-Model Head
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2
Muon motivates designing optimizer geometry around the function of each parameter block and uses the spectral norm for hidden linear layers. For the language-model head, the spectr…
Understanding Adam Requires Better Rotation Dependent Assumptions
Tianyue H. Zhang, Lucas Maes, Alan Milligan +5
Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity t…
Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection
Charles Guille-Escuret, Pierre-André Noël, Ioannis Mitliagkas +2
Improving the reliability of deployed machine learning systems often involves developing methods to detect out-of-distribution (OOD) inputs. However, existing research often narrow…
No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras +1
Understanding the optimization dynamics of neural networks is necessary for closing the gap between theory and practice. Stochastic first-order optimization algorithms are known to…
Towards Out-of-Distribution Adversarial Robustness
Adam Ibrahim, Charles Guille-Escuret, Ioannis Mitliagkas +3
Adversarial robustness continues to be a major challenge for deep learning. A core issue is that robustness to one type of attack often fails to transfer to other attacks. While pr…
CADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning
Charles Guille-Escuret, Pau Rodriguez, David Vazquez +2
Handling out-of-distribution (OOD) samples has become a major stake in the real-world deployment of machine learning systems. This work explores the use of self-supervised contrast…
A Study of Condition Numbers for First-Order Optimization
Charles Guille-Escuret, Baptiste Goujaud, Manuela Girotti +1
The study of first-order optimization algorithms (FOA) typically starts with assumptions on the objective functions, most commonly smoothness and strong convexity. These metrics ar…