6 papers
Amortized Inference of Causal Models via Conditional Fixed-Point Iterations
Divyat Mahajan, Jannes Gladrow, Agrin Hilmkil +2
Structural Causal Models (SCMs) offer a principled framework to reason about interventions and support out-of-distribution generalization, which are key goals in scientific discove…
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
Chao Ma, Wenbo Gong, Meyer Scetbon +1
Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they often require to maintain optimizer states throughout…
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Wenbo Gong, Meyer Scetbon, Chao Ma +1
Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step t…
Gradient Multi-Normalization for Stateless and Scalable LLM Training
Meyer Scetbon, Chao Ma, Wenbo Gong +1
Training large language models (LLMs) typically relies on adaptive optimizers like Adam (Kingma & Ba, 2015) which store additional state information to accelerate convergence but i…
A Fixed-Point Approach for Causal Generative Modeling
Meyer Scetbon, Joel Jennings, Agrin Hilmkil +2
We propose a novel formalism for describing Structural Causal Models (SCMs) as fixed-point problems on causally ordered variables, eliminating the need for Directed Acyclic Graphs…
Low-Rank Correction for Quantized LLMs
Meyer Scetbon, James Hensman
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of ca…