4 papers
Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
Neil Zeghidour, Eugene Kharitonov, Manu Orsini +6
We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an off…
Vision-Speech Models: Teaching Speech Models to Converse about Images
Amélie Royer, Moritz Böhle, Gabriel de Marmiesse +4
The recent successes of Vision-Language models raise the question of how to equivalently imbue a pretrained speech model with vision understanding, an important milestone towards b…
High-Fidelity Simultaneous Speech-To-Speech Translation
Tom Labiausse, Laurent Mazaré, Edouard Grave +3
We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and…
Moshi: a speech-text foundation model for real-time dialogue
Alexandre Défossez, Laurent Mazaré, Manu Orsini +5
We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namel…