13 citations · 23 across the 18 of their papers we have counts for
21 papers
DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
Hyung Gun Chi, Zakaria Aldeneh, Tatiana Likhomanenko +5
We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing dist…
A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Li-Wei Chen, Takuya Higuchi, Zakaria Aldeneh +2
The success of large language models in text processing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it is often discretized f…
Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang +3
Unified decoder-only transformers have shown promise for multimodal generation, yet how sequence design affects synchronization across modalities with heterogeneous sampling rates…
Learning Spatially-Aware Language and Audio Embeddings
Bhavika Devnani, Skyler Seto, Zakaria Aldeneh +5
Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came…
Speaker-IPL: Unsupervised Learning of Speaker Characteristics with i-Vector based Pseudo-Labels
Zakaria Aldeneh, Takuya Higuchi, Jee-weon Jung +6
Iterative self-training, or iterative pseudo-labeling (IPL) -- using an improved model from the current iteration to provide pseudo-labels for the next iteration -- has proven to b…
Exploring Prediction Targets in Masked Pre-Training for Speech Foundation Models
Li-Wei Chen, Takuya Higuchi, He Bai +6
Speech foundation models, such as HuBERT and its variants, are pre-trained on large amounts of unlabeled speech data and then used for a range of downstream tasks. These models use…
Towards Automatic Assessment of Self-Supervised Speech Models using Rank
Zakaria Aldeneh, Vimal Thilak, Takuya Higuchi +2
This study explores using embedding rank as an unsupervised evaluation metric for general-purpose speech encoders trained via self-supervised learning (SSL). Traditionally, assessi…
dMel: Speech Tokenization made Simple
Richard He Bai, Tatiana Likhomanenko, Ruixiang Zhang +3
Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have inv…
Can you Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?
Zakaria Aldeneh, Takuya Higuchi, Jee-weon Jung +6
Self-supervised features are typically used in place of filter-bank features in speaker verification models. However, these models were originally designed to ingest filter-bank fe…
ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
Jee-weon Jung, Wangyou Zhang, Jiatong Shi +5
This paper introduces ESPnet-SPK, a toolkit designed with several objectives for training speaker embedding extractors. First, we provide an open-source platform for researchers in…
Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
Miguel Sarabia, Elena Menyaylenko, Alessandro Toso +7
We present Spatial LibriSpeech, a spatial audio dataset with over 650 hours of 19-channel audio, first-order ambisonics, and optional distractor noise. Spatial LibriSpeech is desig…
Naturalistic Head Motion Generation from Speech
Trisha Mittal, Zakaria Aldeneh, Masha Fedzechkina +2
Synthesizing natural head motion to accompany speech for an embodied conversational agent is necessary for providing a rich interactive experience. Most prior works assess the qual…
On the role of Lip Articulation in Visual Speech Perception
Zakaria Aldeneh, Masha Fedzechkina, Skyler Seto +4
Generating realistic lip motion from audio to simulate speech production is critical for driving natural character animation. Previous research has shown that traditional metrics u…
Aphasic Speech Recognition using a Mixture of Speech Intelligibility Experts
Matthew Perez, Zakaria Aldeneh, Emily Mower Provost
Robust speech recognition is a key prerequisite for semantic feature extraction in automatic aphasic speech analysis. However, standard one-size-fits-all automatic speech recogniti…
On the Role of Visual Cues in Audiovisual Speech Enhancement
Zakaria Aldeneh, Anushree Prasanna Kumar, Barry-John Theobald +4
We present an introspection of an audiovisual speech enhancement model. In particular, we focus on interpreting how a neural audiovisual speech enhancement model uses visual cues t…
Identifying Mood Episodes Using Dialogue Features from Clinical Interviews
Zakaria Aldeneh, Mimansa Jaiswal, Michael Picheny +2
Bipolar disorder, a severe chronic mental illness characterized by pathological mood swings from depression to mania, requires ongoing symptom severity tracking to both guide and m…
Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
Mimansa Jaiswal, Zakaria Aldeneh, Emily Mower Provost
Various psychological factors affect how individuals express emotions. Yet, when we collect data intended for use in building emotion recognition systems, we often try to do so by…
MuSE-ing on the Impact of Utterance Ordering On Crowdsourced Emotion Annotations
Mimansa Jaiswal, Zakaria Aldeneh, Cristian-Paul Bara +4
Emotion recognition algorithms rely on data annotated with high quality labels. However, emotion expression and perception are inherently subjective. There is generally not a singl…
Improving End-of-turn Detection in Spoken Dialogues by Detecting Speaker Intentions as a Secondary Task
Zakaria Aldeneh, Dimitrios Dimitriadis, Emily Mower Provost
This work focuses on the use of acoustic cues for modeling turn-taking in dyadic spoken dialogues. Previous work has shown that speaker intentions (e.g., asking a question, utterin…
Capturing Long-term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition
Soheil Khorram, Zakaria Aldeneh, Dimitrios Dimitriadis +2
The goal of continuous emotion recognition is to assign an emotion value to every frame in a sequence of acoustic features. We show that incorporating long-term temporal dependenci…
Progressive Neural Networks for Transfer Learning in Emotion Recognition
John Gideon, Soheil Khorram, Zakaria Aldeneh +2
Many paralinguistic tasks are closely related and thus representations learned in one domain can be leveraged for another. In this paper, we investigate how knowledge can be transf…