3.8k citations · 7.6k across the 48 of their papers we have counts for
50 papers
Force field optimization by end-to-end differentiable atomistic simulation
Abhijeet S. Gangan, Samuel S. Schoenholz, Ekin Dogus Cubuk +2
The accuracy of atomistic simulations depends on the precision of force fields. Traditional numerical methods often struggle to optimize the empirical force field parameters for re…
Inferring interaction potentials from stochastic particle trajectories
Ella M. King, Megan C. Engel, Caroline Martin +5
Accurate interaction potentials between microscopic components such as colloidal particles or cells are crucial to understanding a range of processes, including colloidal crystalli…
Programmable patchy particles for materials design
Ella M. King, Chrisy Xiyu Du, Qian-Ze Zhu +2
Direct design of complex functional materials would revolutionize technologies ranging from printable organs to novel clean energy devices. However, even incremental steps towards…
What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries
Stanislav Fort, Ekin Dogus Cubuk, Surya Ganguli +1
Deep neural network classifiers partition input space into high confidence regions for each class. The geometry of these class manifolds (CMs) is widely studied and intimately rela…
Deep equilibrium networks are sensitive to initialization statistics
Atish Agarwala, Samuel S. Schoenholz
Deep equilibrium networks (DEQs) are a promising way to construct models which trade off memory for compute. However, theoretical understanding of these models is still lacking com…
Fast Finite Width Neural Tangent Kernel
Roman Novak, Jascha Sohl-Dickstein, Samuel S. Schoenholz
The Neural Tangent Kernel (NTK), defined as where $\left[\partia…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
Gradients are Not All You Need
Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz +1
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are…
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
James Martens, Andy Ballard, Guillaume Desjardins +4
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep n…
Learn2Hop: Learned Optimization on Rough Landscapes
Amil Merchant, Luke Metz, Sam Schoenholz +1
Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material d…
dPV: An End-to-End Differentiable Solar-Cell Simulator
Sean Mann, Eric Fadel, Samuel S. Schoenholz +3
We introduce dPV, an end-to-end differentiable photovoltaic (PV) cell simulator based on the drift-diffusion model and Beer-Lambert law for optical absorption. dPV is programmed in…
Tilting the playing field: Dynamical loss functions for machine learning
Miguel Ruiz-Garcia, Ge Zhang, Samuel S. Schoenholz +1
We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamica…
Self-assembling kinetics: Accessing a new design space via differentiable statistical-physics models
Carl P. Goodrich, Ella M. King, Samuel S. Schoenholz +2
The inverse problem of designing component interactions to target emergent structure is fundamental to numerous applications in biotechnology, materials science, and statistical ph…
Temperature check: theory and practice for training models with softmax-cross-entropy losses
Atish Agarwala, Jeffrey Pennington, Yann Dauphin +1
The softmax function combined with a cross-entropy loss is a principled approach to modeling probability distributions that has become ubiquitous in deep learning. The softmax func…
Unifying framework for strong and fragile liquids via machine learning: a study of liquid silica
Ekin D. Cubuk, Andrea J. Liu, Efthimios Kaxiras +1
The fragility of a glassforming liquid characterizes how rapidly its relaxation dynamics slow down with cooling. The viscosity of strong liquids follows an Arrhenius law with a tem…
Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
Neha S. Wadia, Daniel Duckworth, Samuel S. Schoenholz +2
Machine learning is predicated on the concept of generalization: a model achieving low error on a sufficiently large training set should also perform well on novel samples from the…
Finite Versus Infinite Neural Networks: an Empirical Study
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington +4
We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods. By doing so, we resolve a variety of open que…
On the infinite width limit of neural networks with a standard parameterization
Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz +1
There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the nai…
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Roman Novak, Lechao Xiao, Jiri Hron +4
Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network ar…
Disentangling Trainability and Generalization in Deep Neural Networks
Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz
A longstanding goal in the theory of deep learning is to characterize the conditions under which a given neural network architecture will be trainable, and if so, how well it might…
JAX, M.D.: A Framework for Differentiable Physics
Samuel S. Schoenholz, Ekin D. Cubuk
We introduce JAX MD, a software package for performing differentiable physics simulations with a focus on molecular dynamics. JAX MD includes a number of physics simulation environ…
A Mean Field Theory of Batch Normalization
Greg Yang, Jeffrey Pennington, Vinay Rao +2
We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and…
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz +4
A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have mad…
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Dar Gilboa, Bo Chang, Minmin Chen +4
Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forwa…