5 citations · 19 across the 11 of their papers we have counts for
11 papers
Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product
Mufan Li, Jaume de Dios Pont, Mihai Nica +1
We study the squared singular value spectrum of a non-square product of independent real Gaussian matrices, equivalently the feature covariance spectrum of a deep linear neural net…
Sampling from the Mean-Field Stationary Distribution
Yunbum Kook, Matthew S. Zhang, Sinho Chewi +2
We study the complexity of sampling from the stationary distribution of a mean-field SDE, or equivalently, the complexity of minimizing a functional over the space of probability m…
Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
Mufan Bill Li, Mihai Nica
Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential e…
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Blake Bordelon, Lorenzo Noci, Mufan Bill Li +2
The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such…
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
Lorenzo Noci, Chuning Li, Mufan Bill Li +4
In deep learning theory, the covariance matrix of the representations serves as a proxy to examine the network's trainability. Motivated by the success of Transformers, we study th…
Improved Discretization Analysis for Underdamped Langevin Monte Carlo
Matthew Zhang, Sinho Chewi, Mufan Bill Li +2
Underdamped Langevin Monte Carlo (ULMC) is an algorithm used to sample from unnormalized densities by leveraging the momentum of a particle moving in a potential well. We provide a…
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Mufan Bill Li, Mihai Nica, Daniel M. Roy
The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we…
Acceleration of Gossip Algorithms through the Euler-Poisson-Darboux Equation
Raphaël Berthier, Mufan Bill Li
Gossip algorithms and their accelerated versions have been studied exclusively in discrete time on graphs. In this work, we take a different approach, and consider the scaling limi…
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
Mufan Bill Li, Mihai Nica, Daniel M. Roy
Theoretical results show that neural networks can be approximated by Gaussian processes in the infinite-width limit. However, for fully connected networks, it has been previously s…
Higher Order Generalization Error for First Order Discretization of Langevin Diffusion
Mufan Bill Li, Maxime Gazeau
We propose a novel approach to analyze generalization error for discretizations of Langevin diffusion, such as the stochastic gradient Langevin dynamics (SGLD). For an toleranc…
Riemannian Langevin Algorithm for Solving Semidefinite Programs
Mufan Bill Li, Murat A. Erdogdu
We propose a Langevin diffusion-based algorithm for non-convex optimization and sampling on a product manifold of spheres. Under a logarithmic Sobolev inequality, we establish a gu…