93 citations · 371 across the 20 of their papers we have counts for
18 papers · 1 filter
Gemma 3 Technical Report
Gemma Team, Aishwarya Kamath, Johan Ferret +209
We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision underst…
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin +6
Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional…
AudioPaLM: A Large Language Model That Can Speak and Listen
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen +27
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and…
Generative Spoken Dialogue Language Modeling
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet +8
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled wi…
textless-lib: a Library for Textless Spoken Language Processing
Eugene Kharitonov, Jade Copet, Kushal Lakhotia +8
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this p…
How BPE Affects Memorization in Transformers
Eugene Kharitonov, Marco Baroni, Dieuwke Hupkes
Training data memorization in NLP can both be beneficial (e.g., closed-book QA) and undesirable (personal data extraction). In any case, successful model training requires a non-tr…
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
Felix Kreuk, Adam Polyak, Jade Copet +7
Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the…
Text-Free Prosody-Aware Generative Spoken Language Modeling
Eugene Kharitonov, Ann Lee, Adam Polyak +8
Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragra…
Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN
Rahma Chaabouni, Roberto Dessì, Eugene Kharitonov
Despite their practical success, modern seq2seq architectures are unable to generalize systematically on several SCAN tasks. Hence, it is not clear if SCAN-style compositional gene…
Interpretable agent communication from scratch (with a generic visual processor emerging on the side)
Roberto Dessì, Eugene Kharitonov, Marco Baroni
As deep networks begin to be deployed as autonomous agents, the issue of how they can communicate with each other becomes important. Here, we train two deep nets from scratch to pe…
The Zero Resource Speech Challenge 2021: Spoken language modelling
Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis +6
We present the Zero Resource Speech Challenge 2021, which asks participants to learn a language model directly from audio, without any text or labels. The challenge is based on the…
What they do when in doubt: a study of inductive biases in seq2seq learners
Eugene Kharitonov, Rahma Chaabouni
Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by inve…
Emergent Language Generalization and Acquisition Speed are not tied to Compositionality
Eugene Kharitonov, Marco Baroni
Studies of discrete languages emerging when neural agents communicate to solve a joint task often look for evidence of compositional structure. This stems for the expectation that…
Compositionality and Generalization in Emergent Languages
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt +2
Natural language allows us to refer to novel composite concepts by combining expressions denoting their parts according to systematic rules, a property known as \emph{compositional…
EGG: a toolkit for research on Emergence of lanGuage in Games
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt +1
There is renewed interest in simulating language emergence among deep neural agents that communicate to jointly solve a task, spurred by the practical aim to develop language-enabl…
Word-order biases in deep-agent emergent communication
Rahma Chaabouni, Eugene Kharitonov, Alessandro Lazaric +2
Sequence-processing neural networks led to remarkable progress on many NLP tasks. As a consequence, there has been increasing interest in understanding to what extent they process…
Entropy Minimization In Emergent Languages
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt +1
There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investi…
Anti-efficient encoding in emergent communication
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux +1
Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human langua…