98 citations · 320 across the 13 of their papers we have counts for
13 papers
Mechanistic Mode Connectivity
Ekdeep Singh Lubana, Eric J. Bigelow, Robert P. Dick +2
We study neural network loss landscapes through the lens of mode connectivity, the observation that minimizers of neural networks retrieved via training on a dataset are connected…
What shapes the loss landscape of self-supervised learning?
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda +1
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theor…
The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie +4
In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD). As observed previously, long after performance has converged,…
Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning
Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to expla…
Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
Hidenori Tanaka, Daniel Kunin
In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture…
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli +2
Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is t…
Pruning neural networks without any data by iteratively conserving synaptic flow
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins +1
Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent work…
From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan +3
Recently, deep feedforward neural networks have achieved considerable success in modeling biological sensory processing, in terms of reproducing the input-output map of sensory neu…
Non-Hermitian Quasi-Localization and Ring Attractor Neural Networks
Hidenori Tanaka, David R. Nelson
Eigenmodes of a broad class of "sparse" random matrices, with interactions concentrated near the diagonal, exponentially localize in space, as initially discovered in 1957 by Ander…
Spatial gene drives and pushed genetic waves
Hidenori Tanaka, Howard A. Stone, David R. Nelson
Gene drives have the potential to rapidly replace a harmful wild-type allele with a gene drive allele engineered to have desired functionalities. However, an accidental or prematur…
Hot Particles Attract in a Cold Bath
Hidenori Tanaka, Alpha A. Lee, Michael P. Brenner
Controlling interactions out of thermodynamic equilibrium is crucial for designing addressable and functional self-organizing structures. These active interactions also underpin co…
Mutation at Expanding Front of Self-Replicating Colloidal Clusters
Hidenori Tanaka, Zorana Zeravcic, Michael P. Brenner
We construct a scheme for self-replicating square clusters of particles in two spatial dimensions, and validate it with computer simulations in a finite-temperature heat bath. We f…
Quenched metastable vortex states in Sr2RuO4
Daisuke Shibata, Hidenori Tanaka, Shingo Yonezawa +2
Sr2RuO4, a leading-candidate spin-triplet superconductor and a highly anisotropic quasi-two-dimensional type-II superconductor, provides unique opportunity to study unconventional…