138 citations · 310 across the 39 of their papers we have counts for
40 papers
Learning the identity: a case study of how SGD selects among functional decompositions
Andy Arditi, Weian Xie, David Bau +1
One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the…
Thermodynamic Irreversibility of Training Algorithms
Liu Ziyin, Yuanjie Ren, Adam Levine +1
The training algorithms for AI systems all introduce far-from-equilibrium dynamical processes, and understanding the irreversibility of these algorithms is a fundamental step towar…
On the Invariance and Generality of Neural Scaling Laws
Xing Han, Ziyin Liu, Liu Ziyin +2
Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks.…
Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model
Yizhou Xu, Pierfrancesco Beneventano, Isaac Chuang +1
A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there ha…
Topological Invariance and Breakdown in Learning
Yongyi Yang, Tomaso Poggio, Isaac Chuang +1
We prove that for a broad class of permutation-equivariant learning rules (including SGD, Adam, and others), the training process induces a bi-Lipschitz mapping between neurons and…
Proof of a perfect platonic representation hypothesis
Liu Ziyin, Isaac Chuang
In this note, we elaborate on and explain in detail the proof given by Ziyin et al. (2025) of the ``perfect" Platonic Representation Hypothesis (PRH) for the embedded deep linear n…
Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Liu Ziyin, Yizhou Xu, Isaac Chuang
With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic…
Understanding the Emergence of Multimodal Representation Alignment
Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin +1
Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly…
Parameter Symmetry Potentially Unifies Deep Learning Theory
Liu Ziyin, Yizhou Xu, Tomaso Poggio +1
The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While…
How Biological Synapses Self-Assemble Gradient Learning
Qianli Liao, Liu Ziyin, Yulu Gan +4
Existing models of learning in the brain explain how given circuits learn, but not how biology could assemble those circuits in the first place. To address this gap, we formulate S…
Formation of Representations in Neural Networks
Liu Ziyin, Isaac Chuang, Tomer Galanti +1
Understanding neural representations will help open the black box of neural networks and advance our scientific understanding of modern AI systems. However, how complex, structured…
Remove Symmetries to Control Model Expressivity and Improve Optimization
Liu Ziyin, Yizhou Xu, Isaac Chuang
When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a "collapse". Being trapped in these low-capacit…
Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
Liu Ziyin, Mingze Wang, Hongchao Li +1
Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries -- a broad…
Three Mechanisms of Feature Learning in a Linear Network
Yizhou Xu, Liu Ziyin
Understanding the dynamics of neural networks in different width regimes is crucial for improving their training and performance. We present an exact solution for the learning dyna…
Symmetry Induces Structure and Constraint of Learning
Liu Ziyin
Due to common architecture designs, symmetries exist extensively in contemporary neural networks. In this work, we unveil the importance of the loss function symmetries in affectin…
Noise Balance and Stationary Distribution of Stochastic Gradient Descent
Liu Ziyin, Hongchao Li, Masahito Ueda
The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks. However, it remains poorly understood how the SGD navigates the highly nonlinear a…
On the Stepwise Nature of Self-Supervised Learning
James B. Simon, Maksis Knutins, Liu Ziyin +3
We present a simple picture of the training process of joint embedding self-supervised learning methods. We find that these methods learn their high-dimensional embeddings one dime…
Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent
Liu Ziyin, Botao Li, Tomer Galanti +1
Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks…
spred: Solving Penalty with SGD
Liu Ziyin, Zihao Wang
We propose to minimize a generic differentiable objective with constraint using a simple reparametrization and straightforward stochastic gradient descent. Our proposal is th…
What shapes the loss landscape of self-supervised learning?
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda +1
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theor…
Exact Phase Transitions in Deep Learning
Liu Ziyin, Masahito Ueda
This work reports deep-learning-unique first-order and second-order phase transitions, whose phenomenology closely follows that in statistical physics. In particular, we prove that…
Posterior Collapse of a Linear Latent Variable Model
Zihao Wang, Liu Ziyin
This work identifies the existence and cause of a type of posterior collapse that frequently occurs in the Bayesian deep learning practice. For a general linear latent variable mod…
Universal Thermodynamic Uncertainty Relation in Non-Equilibrium Dynamics
Liu Ziyin, Masahito Ueda
We derive a universal thermodynamic uncertainty relation (TUR) that applies to an arbitrary observable in a general Markovian system. The generality of our result allows us to make…
Exact Solutions of a Deep Linear Network
Liu Ziyin, Botao Li, Xiangming Meng
This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscap…