16 papers
Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics
Jihwan Lee, Tiantian Feng, Minu Kim +6
Neural audio codecs are increasingly used for low-bitrate speech coding and serve as tokenizers for speech language models, yet it remains unclear how their codebook hierarchies en…
Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
Hong Nguyen, Sean Foley, Christina Hagedorn +4
Real-time MRI (rtMRI) captures the dynamics of the entire vocal tract during speech, but labeled data are scarce and the modality - single-slice, grayscale, low-resolution - differ…
Learning-free L2-Accented Speech Generation using Phonological Rules
Thanathai Lertpetchpun, Yoonjeong Lee, Jihwan Lee +3
Accent plays a crucial role in speaker identity and inclusivity in speech technologies. Existing accented text-to-speech (TTS) systems either require large-scale accented datasets…
Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data
Thanathai Lertpetchpun, Thanapat Trachu, Jihwan Lee +3
Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet…
Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis
Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu +4
Many spoken languages, including English, exhibit wide variation in dialects and accents, making accent control an important capability for flexible text-to-speech (TTS) models. Cu…
A Point Process Model of Skin Conductance Responses in a Stroop Task for Predicting Depression and Suicidal Ideation
Kleanthis Avramidis, Myzelle Hughes, Idan A Blank +5
Accurate identification of mental health biomarkers can enable earlier detection and objective assessment of compromised mental well-being. In this study, we analyze electrodermal…
ARTI-6: Towards Six-dimensional Articulatory Speech Encoding
Jihwan Lee, Sean Foley, Thanathai Lertpetchpun +6
We propose ARTI-6, a compact six-dimensional articulatory speech encoding framework derived from real-time MRI data that captures crucial vocal tract regions including the velum, t…
Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition
Jay Park, Hong Nguyen, Sean Foley +4
Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and…
Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe
Tiantian Feng, Kevin Huang, Anfeng Xu +6
We present Voxlect, a novel benchmark for modeling dialects and regional languages worldwide using speech foundation models. Specifically, we report comprehensive benchmark evaluat…
On the Relationship between Accent Strength and Articulatory Features
Kevin Huang, Sean Foley, Jihwan Lee +3
This paper explores the relationship between accent strength and articulatory features inferred from acoustic speech. To quantify accent strength, we compare phonetic transcription…
Time-Resolved EEG Decoding of Semantic Processing Reveals Altered Neural Dynamics in Depression and Suicidality
Woojae Jeong, Aditya Kommineni, Kleanthis Avramidis +15
Depression and suicidality affect cognitive and emotional processes, yet objective, task-evoked neural readouts of mental health remain limited. We investigated the spatiotemporal…
Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction
Thanathai Lertpetchpun, Tiantian Feng, Dani Byrd +1
Speech emotion recognition (SER) in naturalistic conditions presents a significant challenge for the speech processing community. Challenges include disagreement in labeling among…
Neural Responses to Affective Sentences Reveal Signatures of Depression
Aditya Kommineni, Woojae Jeong, Kleanthis Avramidis +13
Major Depressive Disorder (MDD) is a highly prevalent mental health condition, and a deeper understanding of its neurocognitive foundations is essential for identifying how core fu…
Developing a Top-tier Framework in Naturalistic Conditions Challenge for Categorized Emotion Prediction: From Speech Foundation Models and Learning Objective to Data Augmentation and Engineering Choices
Tiantian Feng, Thanathai Lertpetchpun, Dani Byrd +1
Speech emotion recognition (SER), particularly for naturally expressed emotions, remains a challenging computational task. Key challenges include the inherent subjectivity in emoti…
Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties
Tiantian Feng, Jihwan Lee, Anfeng Xu +9
We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a singl…
Deep Learning Characterizes Depression and Suicidal Ideation from Eye Movements
Kleanthis Avramidis, Woojae Jeong, Aditya Kommineni +14
Identifying physiological and behavioral markers for mental health conditions is a longstanding challenge in psychiatry. Depression and suicidal ideation, in particular, lack objec…