1 citations · 2 across the 23 of their papers we have counts for
23 papers
High-Dimensional Learning Dynamics of Attention-Indexed Models
Yizhou Xu, Margarita Sagitova, Lenka Zdeborová +1
Attention mechanisms are central to modern foundation models, yet their training dynamics remain poorly understood, especially when the attention matrices have extensive rank. In t…
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Liu Ziyin, Yizhou Xu, Tomaso Poggio +1
Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training…
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Julius Girardin, Emanuele Troiani, Yizhou Xu +3
Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe…
Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model
Yizhou Xu, Pierfrancesco Beneventano, Isaac Chuang +1
A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there ha…
PromptTailor: Multi-turn Intent-Aligned Prompt Synthesis for Lightweight LLMs
Yizhou Xu, Janet Davis
Lightweight language models remain attractive for on-device and privacy-sensitive applications, but their responses are highly sensitive to prompt quality. For open-ended generatio…
Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning
Rongrong Xie, Yizhou Xu, Guido Sanguinetti
The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer "teacher" modalities transfe…
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Leonardo Defilippis, Yizhou Xu, Julius Girardin +5
Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a…
Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
Yizhou Xu, Florent Krzakala, Lenka Zdeborová
The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its perfor…
Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Liu Ziyin, Yizhou Xu, Isaac Chuang
With the rapid discovery of emergent phenomena in deep learning and large language models, understanding their cause has become an urgent need. Here, we propose a rigorous entropic…
Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applications
Yizhou Xu, Antoine Maillard, Lenka Zdeborová +1
In the matrix sensing problem, one wishes to reconstruct a matrix from (possibly noisy) observations of its linear projections along given directions. We consider this model in the…
Parameter Symmetry Potentially Unifies Deep Learning Theory
Liu Ziyin, Yizhou Xu, Tomaso Poggio +1
The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While…
The Error Probability of Spatially Coupled Sparse Regression Codes over Memoryless Channels
Yuhao Liu, Yizhou Xu, Tianqi Hou
Sparse Regression Codes (SPARCs) are capacity-achieving codes introduced for communication over the Additive White Gaussian Noise (AWGN) channels and were later extended to general…
Fluid-network relations: decay laws meet with spatial self-similarity, scale-invariance, and control scaling
Yang Tian, Pei Sun, Yizhou Xu
Diverse implicit structures of fluids are discovered lately, providing opportunities to study the physics of fluids applying network analysis. Although considerable works devote to…
Remove Symmetries to Control Model Expressivity and Improve Optimization
Liu Ziyin, Yizhou Xu, Isaac Chuang
When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a "collapse". Being trapped in these low-capacit…
Information limits and Thouless-Anderson-Palmer equations for spiked matrix models with structured noise
Jean Barbier, Francesco Camilli, Marco Mondelli +1
We consider a prototypical problem of Bayesian inference for a structured spiked model: a low-rank signal is corrupted by additive noise. While both information-theoretic and algor…
Decentralizing Coherent Joint Transmission Precoding via Fast ADMM with Deterministic Equivalents
Xinyu Bian, Yuhao Liu, Yizhou Xu +4
Inter-cell interference (ICI) suppression is critical for multi-cell multi-user networks. In this paper, we investigate advanced precoding techniques for coordinated multi-point (C…
Decentralizing Coherent Joint Transmission Precoding via Deterministic Equivalents
Yuhao Liu, Xinyu Bian, Yizhou Xu +4
In order to control the inter-cell interference for a multi-cell multi-user multiple-input multiple-output network, we consider the precoder design for coordinated multi-point with…
Fast renormalizing the structures and dynamics of ultra-large systems via random renormalization group
Yang Tian, Yizhou Xu, Pei Sun
Criticality and symmetry, studied by the renormalization groups, lie at the heart of modern physics theories of matters and complex systems. However, surveying these properties wit…
Three Mechanisms of Feature Learning in a Linear Network
Yizhou Xu, Liu Ziyin
Understanding the dynamics of neural networks in different width regimes is crucial for improving their training and performance. We present an exact solution for the learning dyna…
Capacity-Achieving Sparse Regression Codes via Vector Approximate Message Passing
Yizhou Xu, YuHao Liu, ShanSuo Liang +4
Sparse regression codes (SPARCs) are a promising coding scheme that can approach the Shannon limit over Additive White Gaussian Noise (AWGN) channels. Previous works have proven th…
Thermodynamics of percolation in interacting systems
Yizhou Xu, Pei Sun, Yang Tian
Interacting systems can be studied as the networks where nodes are system units and edges denote correlated interactions. Although percolation on network is a unified way to model…
Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Models
Yizhou Xu, TianQi Hou, ShanSuo Liang +1
We consider the problem of reconstructing the signal and the hidden variables from observations coming from a multi-layer network with rotationally invariant weight matrices. The m…
Adversarial Jamming for a More Effective Constellation Attack
Haidong Xie, Yizhou Xu, Yuanqing Chen +4
The common jamming mode in wireless communication is band barrage jamming, which is controllable and difficult to resist. Although this method is simple to implement, it is obvious…