13 citations · 46 across the 21 of their papers we have counts for
22 papers
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Wenbo Gong, Meyer Scetbon, Chao Ma +1
Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step t…
Gradient Multi-Normalization for Stateless and Scalable LLM Training
Meyer Scetbon, Chao Ma, Wenbo Gong +1
Training large language models (LLMs) typically relies on adaptive optimizers like Adam (Kingma & Ba, 2015) which store additional state information to accelerate convergence but i…
Low-Rank Correction for Quantized LLMs
Meyer Scetbon, James Hensman
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of ca…
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
Chao Ma, Wenbo Gong, Meyer Scetbon +1
Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they often require to maintain optimizer states throughout…
Amortized Inference of Causal Models via Conditional Fixed-Point Iterations
Divyat Mahajan, Jannes Gladrow, Agrin Hilmkil +2
Structural Causal Models (SCMs) offer a principled framework to reason about interventions and support out-of-distribution generalization, which are key goals in scientific discove…
A Fixed-Point Approach for Causal Generative Modeling
Meyer Scetbon, Joel Jennings, Agrin Hilmkil +2
We propose a novel formalism for describing Structural Causal Models (SCMs) as fixed-point problems on causally ordered variables, eliminating the need for Directed Acyclic Graphs…
The Essential Role of Causality in Foundation World Models for Embodied AI
Tarun Gupta, Wenbo Gong, Chao Ma +11
Recent advances in foundation models, especially in large multi-modal models and conversational agents, have ignited interest in the potential of generally capable embodied agents.…
Robust Linear Regression: Phase-Transitions and Precise Tradeoffs for General Norms
Elvis Dohmatob, Meyer Scetbon
In this paper, we investigate the impact of test-time adversarial attacks on linear regression models and determine the optimal level of robustness that any model can reach while m…
Unbalanced Low-rank Optimal Transport Solvers
Meyer Scetbon, Michal Klein, Giovanni Palla +1
The relevance of optimal transport methods to machine learning has long been hindered by two salient limitations. First, the computational cost of standard sample-based so…
Polynomial-Time Solvers for the Discrete -Optimal Transport Problems
Meyer Scetbon
In this note, we propose polynomial-time algorithms solving the Monge and Kantorovich formulations of the -optimal transport problem in the discrete and finite setting. It…
Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond
Meyer Scetbon, Elvis Dohmatob
In this work we study the robustness to adversarial attacks, of early-stopping strategies on gradient-descent (GD) methods for linear regression. More precisely, we show that early…
Low-rank Optimal Transport: Approximation, Statistics and Debiasing
Meyer Scetbon, Marco Cuturi
The matching principles behind optimal transport (OT) play an increasingly important role in machine learning, a trend which can be observed when OT is used to disambiguate dataset…
Triangular Flows for Generative Modeling: Statistical Consistency, Smoothness Classes, and Fast Rates
Nicholas J. Irons, Meyer Scetbon, Soumik Pal +1
Triangular flows, also known as Knöthe-Rosenblatt measure couplings, comprise an important building block of normalizing flow models for generative modeling and density estimation,…
An Asymptotic Test for Conditional Independence using Analytic Kernel Embeddings
Meyer Scetbon, Laurent Meunier, Yaniv Romano
We propose a new conditional dependence measure and a statistical test for conditional independence. The measure is based on the difference between analytic kernel embeddings of tw…
Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs
Meyer Scetbon, Gabriel Peyré, Marco Cuturi
The ability to align points across two related yet incomparable point clouds (e.g. living in different spaces) plays an important role in machine learning. The Gromov-Wasserstein (…
Low-Rank Sinkhorn Factorization
Meyer Scetbon, Marco Cuturi, Gabriel Peyré
Several recent applications of optimal transport (OT) theory to machine learning have relied on regularization, notably entropy and the Sinkhorn algorithm. Because matrix-vector pr…
Mixed Nash Equilibria in the Adversarial Examples Game
Laurent Meunier, Meyer Scetbon, Rafael Pinot +2
This paper tackles the problem of adversarial examples from a game theoretic point of view. We study the open question of the existence of mixed Nash equilibria in the zero-sum gam…
Equitable and Optimal Transport with Multiple Agents
Meyer Scetbon, Laurent Meunier, Jamal Atif +1
We introduce an extension of the Optimal Transport problem when multiple costs are involved. Considering each cost as an agent, we aim to share equally between agents the work of t…
Linear Time Sinkhorn Divergences using Positive Features
Meyer Scetbon, Marco Cuturi
Although Sinkhorn divergences are now routinely used in data sciences to compare probability distributions, the computational effort required to compute them remains expensive, gro…
Harmonic Decompositions of Convolutional Networks
Meyer Scetbon, Zaid Harchaoui
We present a description of the function space and the smoothness class associated with a convolutional network using the machinery of reproducing kernel Hilbert spaces. We show th…
A Spectral Analysis of Dot-product Kernels
Meyer Scetbon, Zaid Harchaoui
We present eigenvalue decay estimates of integral operators associated with compositional dot-product kernels. The estimates improve on previous ones established for power series k…
Deep K-SVD Denoising
Meyer Scetbon, Michael Elad, Peyman Milanfar
This work considers noise removal from images, focusing on the well known K-SVD denoising algorithm. This sparsity-based method was proposed in 2006, and for a short while it was c…