98 citations · 216 across the 13 of their papers we have counts for
6 papers · 1 filter
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Wenbo Gong, Meyer Scetbon, Chao Ma +1
Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step t…
Gradient Multi-Normalization for Stateless and Scalable LLM Training
Meyer Scetbon, Chao Ma, Wenbo Gong +1
Training large language models (LLMs) typically relies on adaptive optimizers like Adam (Kingma & Ba, 2015) which store additional state information to accelerate convergence but i…
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
Chao Ma, Wenbo Gong, Meyer Scetbon +1
Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they often require to maintain optimizer states throughout…
Deterministic Variational Inference for Robust Bayesian Neural Networks
Anqi Wu, Sebastian Nowozin, Edward Meeds +3
Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probab…
Optimization Monte Carlo: Efficient and Embarrassingly Parallel Likelihood-Free Inference
Edward Meeds, Max Welling
We describe an embarrassingly parallel, anytime Monte Carlo method for likelihood-free models. The algorithm starts with the view that the stochasticity of the pseudo-samples gener…
GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation
Edward Meeds, Max Welling
Scientists often express their understanding of the world through a computationally demanding simulation program. Analyzing the posterior distribution of the parameters given obser…