4 papers
A Fourier Space Perspective on Diffusion Models
Fabian Falck, Teodora Pandeva, Kiarash Zahirnia +5
Diffusion models are state-of-the-art generative models on data modalities such as images, audio, proteins and materials. These modalities share the property of exponentially decay…
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
Chao Ma, Wenbo Gong, Meyer Scetbon +1
Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they often require to maintain optimizer states throughout…
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Wenbo Gong, Meyer Scetbon, Chao Ma +1
Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step t…
Gradient Multi-Normalization for Stateless and Scalable LLM Training
Meyer Scetbon, Chao Ma, Wenbo Gong +1
Training large language models (LLMs) typically relies on adaptive optimizers like Adam (Kingma & Ba, 2015) which store additional state information to accelerate convergence but i…