11 citations · 32 across the 8 of their papers we have counts for
9 papers
Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi +4
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitat…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attacks
Badr Youbi Idrissi, Monica Millunzi, Amelia Sorrenti +2
In the present-day scenario, Large Language Models (LLMs) are establishing their presence as powerful instruments permeating various sectors of society. While their utility offers…
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite +3
Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and simil…
Better & Faster Large Language Models via Multi-token Prediction
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière +2
Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens…
ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero +7
Deep learning vision systems are widely deployed across applications where reliability is critical. However, even today's best models can fail to recognize an object when its pose,…
Simple data balancing achieves competitive worst-group-accuracy
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki +1
We study the problem of learning classifiers that perform well across (known or unknown) groups of data. After observing that common worst-group-accuracy datasets suffer from subst…
Masked Adversarial Generation for Neural Machine Translation
Badr Youbi Idrissi, Stéphane Clinchant
Attacking Neural Machine Translation models is an inherently combinatorial task on discrete sequences, solved with approximate heuristics. Most methods use the gradient to attack t…
VirAAL: Virtual Adversarial Active Learning For NLU
Gregory Senay, Badr Youbi Idrissi, Marine Haziza
This paper presents VirAAL, an Active Learning framework based on Adversarial Training. VirAAL aims to reduce the effort of annotation in Natural Language Understanding (NLU). VirA…