69 citations · 319 across the 14 of their papers we have counts for
24 papers
What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries
Stanislav Fort, Ekin Dogus Cubuk, Surya Ganguli +1
Deep neural network classifiers partition input space into high confidence regions for each class. The geometry of these class manifolds (CMs) is widely studied and intimately rela…
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
James Martens, Andy Ballard, Guillaume Desjardins +4
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep n…
Learn2Hop: Learned Optimization on Rough Landscapes
Amil Merchant, Luke Metz, Sam Schoenholz +1
Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material d…
Tilting the playing field: Dynamical loss functions for machine learning
Miguel Ruiz-Garcia, Ge Zhang, Samuel S. Schoenholz +1
We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamica…
Self-assembling kinetics: Accessing a new design space via differentiable statistical-physics models
Carl P. Goodrich, Ella M. King, Samuel S. Schoenholz +2
The inverse problem of designing component interactions to target emergent structure is fundamental to numerous applications in biotechnology, materials science, and statistical ph…
Temperature check: theory and practice for training models with softmax-cross-entropy losses
Atish Agarwala, Jeffrey Pennington, Yann Dauphin +1
The softmax function combined with a cross-entropy loss is a principled approach to modeling probability distributions that has become ubiquitous in deep learning. The softmax func…