425 citations · 1.2k across the 64 of their papers we have counts for
71 papers
Benchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units
Robin San Roman, Manel Khentout, Tu Anh Nguyen +3
Representation learning has attracted great atten- tion and managed to reach good performances as a pretraining method for downstream tasks or as a first step towards unsu- pervise…
InvFlowFD: Reference-Free and Background-Set-Free Perceptual Music Quality Metric with Flow Matching Inversion
Alon Ziv, Harel Pogoda, Yossi Adi
Existing reference-free methods for evaluating music perceptual quality alleviate the need for paired noisy-clean data, but they still rely on a background set, which is used to co…
Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark
Arnon Turetzky, Avihu Dekel, Hagai Aronowitz +2
Spoken meaning often depends not only on what is said, but also on which word is emphasized. The same sentence can convey correction, contrast, or clarification depending on where…
MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation
Alon Ziv, Sanyuan Chen, Andros Tjandra +3
A key challenge in music generation models is their lack of direct alignment with human preferences, as music evaluation is inherently subjective and varies widely across individua…
PAST: Phonetic-Acoustic Speech Tokenizer
Nadav Har-Tuv, Or Tal, Yossi Adi
We present PAST, a novel end-to-end framework that jointly models phonetic information alongside signal reconstruction, eliminating the need for external pretrained models. Unlike…
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition
Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay +2
We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses…
On The Landscape of Spoken Language Models: A Comprehensive Survey
Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien +7
The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act a…
Scaling Analysis of Interleaved Speech-Text Language Models
Gallil Maimon, Michael Hassid, Amit Roth +1
Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. It predicts that SLMs require much more compute and data compared to text, leading some to question th…
Slamming: Training a Speech Language Model on One GPU in a Day
Gallil Maimon, Avishai Elmakies, Yossi Adi
We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisatio…
Unsupervised Speech Segmentation: A General Approach Using Speech Language Models
Avishai Elmakies, Omri Abend, Yossi Adi
In this paper, we introduce an unsupervised approach for Speech Segmentation, which builds on previously researched approaches, e.g., Speaker Diarization, while being applicable to…
MusicGen-Stem: Multi-stem music generation and edition through autoregressive modeling
Simon Rouard, Robin San Roman, Yossi Adi +1
While most music generation models generate a mixture of stems (in mono or stereo), we propose to train a multi-stem generative model with 3 stems (bass, drums and other) that lear…
Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation
Guy Yariv, Yuval Kirstain, Amit Zohar +5
We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description. While recent advan…
Formal Language Knowledge Corpus for Retrieval Augmented Generation
Majd Zayyad, Yossi Adi
The integration of retrieval-augmented techniques with LLMs has shown promise in improving performance across various domains. However, their utility in tasks requiring advanced re…
Enhancing TTS Stability in Hebrew using Discrete Semantic Units
Ella Zeldes, Or Tal, Yossi Adi
This study introduces a refined approach to Text-to-Speech (TTS) generation that significantly enhances sampling stability across languages, with a particular focus on Hebrew. By l…
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin +6
Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional…
LAST: Language Model Aware Speech Tokenization
Arnon Turetzky, Yossi Adi
Speech tokenization serves as the foundation of speech language model (LM), enabling them to perform various tasks such as spoken language modeling, text-to-speech, speech-to-text,…
Latent Watermarking of Audio Generative Models
Robin San Roman, Pierre Fernandez, Antoine Deleforge +2
The advancements in audio generative models have opened up new challenges in their responsible disclosure and the detection of their misuse. In response, we introduce a method to w…
Salmon: A Suite for Acoustic Language Model Evaluation
Gallil Maimon, Amit Roth, Yossi Adi
Speech language models have recently demonstrated great potential as universal speech processing systems. Such models have the ability to model the rich acoustic information existi…
Audio Enhancement from Multiple Crowdsourced Recordings: A Simple and Effective Baseline
Shiran Aziz, Yossi Adi, Shmuel Peleg
With the popularity of cellular phones, events are often recorded by multiple devices from different locations and shared on social media. Several different recordings could be fou…
Audio Conditioning for Music Generation via Discrete Bottleneck Features
Simon Rouard, Yossi Adi, Jade Copet +2
While most music generation models use textual or parametric conditioning (e.g. tempo, harmony, musical genre), we propose to condition a language model based music generation syst…
A Language Modeling Approach to Diacritic-Free Hebrew TTS
Amit Roth, Arnon Turetzky, Yossi Adi
We tackle the task of text-to-speech (TTS) in Hebrew. Traditional Hebrew contains Diacritics, which dictate the way individuals should pronounce given words, however, modern Hebrew…
HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
Arnon Turetzky, Or Tal, Yael Segal-Feldman +9
We present HebDB, a weakly supervised dataset for spoken language processing in the Hebrew language. HebDB offers roughly 2500 hours of natural and spontaneous speech recordings in…
Discrete Flow Matching
Itai Gat, Tal Remez, Neta Shaul +5
Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional…
NAST: Noise Aware Speech Tokenization for Speech Language Models
Shoval Messica, Yossi Adi
Speech tokenization is the task of representing speech signals as a sequence of discrete units. Such representations can be later used for various downstream tasks including automa…