8 papers
Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
Sam Buchanan, Druv Pai, Peng Wang +1
In the current era of deep learning and especially generative models, there is significant investment in training very large deep neural networks. Thus far, such models have been "…
A Global Geometric Analysis of Maximal Coding Rate Reduction
Peng Wang, Huikang Liu, Druv Pai +4
The maximal coding rate reduction (MCR) objective for learning structured and compact deep representations is drawing increasing attention, especially after its recent usage in…
On the Edge of Memorization in Diffusion Models
Sam Buchanan, Druv Pai, Yi Ma +1
When do diffusion models reproduce their training data, and when are they able to generate samples beyond it? A practically relevant theoretical understanding of this interplay bet…
Attention-Only Transformers via Unrolled Subspace Denoising
Peng Wang, Yifu Lu, Yaodong Yu +3
Despite the popularity of transformers in practice, their architectures are empirically designed and neither mathematically justified nor interpretable. Moreover, as indicated by m…
Independent and Decentralized Learning in Markov Potential Games
Chinmay Maheshwari, Manxi Wu, Druv Pai +1
We study a multi-agent reinforcement learning dynamics, and analyze its asymptotic behavior in infinite-horizon discounted Markov potential games. We focus on the independent and d…
Simplifying DINO via Coding Rate Regularization
Ziyang Wu, Jingyuan Zhang, Druv Pai +5
DINO and DINOv2 are two model families being widely used to learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-t…