354 citations · 1.1k across the 21 of their papers we have counts for
46 papers
Better & Faster Large Language Models via Multi-token Prediction
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière +2
Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens…
Audio Language Modeling using Perceptually-Guided Discrete Representations
Felix Kreuk, Yaniv Taigman, Adam Polyak +4
In this work, we study the task of Audio Language Modeling, in which we aim at learning probabilistic models for audio that can be used for generation and completion. We use a stat…
High Fidelity Neural Audio Compression
Alexandre Défossez, Jade Copet, Gabriel Synnaeve +1
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent spac…
Star Temporal Classification: Sequence Classification with Partially Labeled Data
Vineel Pratap, Awni Hannun, Gabriel Synnaeve +1
We develop an algorithm which can learn from partially labeled and unsegmented sequential data. Most sequential loss functions, such as Connectionist Temporal Classification (CTC),…
Hierarchical Skills for Efficient Exploration
Jonas Gehring, Gabriel Synnaeve, Andreas Krause +1
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike…
Word Order Does Not Matter For Speech Recognition
Vineel Pratap, Qiantong Xu, Tatiana Likhomanenko +2
In this paper, we study training of automatic speech recognition system in a weakly supervised setting where the order of words in transcript labels of the audio training data is n…