3 citations · 10 across the 13 of their papers we have counts for
5 papers · 2 filters
Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
Mathieu Blondel, Michael E. Sander, Germain Vivier-Ardisson +2
Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have h…
Learning with Local Search MCMC Layers
Germain Vivier-Ardisson, Mathieu Blondel, Axel Parmentier
Integrating combinatorial optimization layers into neural networks has recently attracted significant research interest. However, many existing approaches lack theoretical guarante…
On Teacher Hacking in Language Model Distillation
Daniil Tiapkin, Daniele Calandriello, Johan Ferret +4
Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii)…
Loss Functions and Operators Generated by f-Divergences
Vincent Roulet, Tianlin Liu, Nino Vieillard +2
The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token pr…
Joint Learning of Energy-based Models and their Partition Function
Michael E. Sander, Vincent Roulet, Tianlin Liu +1
Energy-based models (EBMs) offer a flexible framework for parameterizing probability distributions using neural networks. However, learning EBMs by exact maximum likelihood estimat…