7 citations · 18 across the 41 of their papers we have counts for
46 papers
Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics
Jihwan Lee, Tiantian Feng, Minu Kim +6
Neural audio codecs are increasingly used for low-bitrate speech coding and serve as tokenizers for speech language models, yet it remains unclear how their codebook hierarchies en…
The Geometry of Lee-Yang Tensors
Jake Hofgard
We provide a geometric description of Lee-Yang tensors, building upon several recent results of Bravyi, Gosset, Liu, and Wong. By applying tools from complex Perron-Frobenius theor…
Speaker Role and Language Diarization for Analyzing Multilingual Interviews for Language Proficiency of Older Adults
Anfeng Xu, Tiantian Feng, Kevin Huang +8
Automatic language proficiency assessment in the context of multilingual interview-based settings remains underexplored. In this work, we develop Whisper-based speaker-role and lan…
Speech Entrainment in Multi-Party Conversations with a Digital Agent
Nicholas Mehlman, Kaitlin Zareno, Kleanthis Avramidis +2
It has been widely observed that individuals engaged in conversation tend to adapt their speaking style to more closely match the other interlocutors. However, most prior work has…
RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2
Respiratory diseases remain a leading cause of global mortality, where timely and accurate diagnosis is critical to improving patient outcomes and reducing healthcare burdens. Whil…
Affect Decoding in Phonated and Silent Speech Production from Surface EMG
Simon Pistrosch, Kleanthis Avramidis, Zhao Ren +7
The expression of affect is integral to spoken communication, yet, its link to underlying articulatory execution remains unclear. Measures of articulatory muscle activity such as E…
Learning-free L2-Accented Speech Generation using Phonological Rules
Thanathai Lertpetchpun, Yoonjeong Lee, Jihwan Lee +3
Accent plays a crucial role in speaker identity and inclusivity in speech technologies. Existing accented text-to-speech (TTS) systems either require large-scale accented datasets…
Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data
Thanathai Lertpetchpun, Thanapat Trachu, Jihwan Lee +3
Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet…
Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs
Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +3
Existing visual privacy benchmarks largely treat privacy as a binary property, labeling images as private or non-private based on visible sensitive content. We argue that privacy i…
TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild
Kai-Wei Chang, Yi-Cheng Lin, Huang-Cheng Chou +9
Speech technologies have advanced rapidly and serve diverse populations worldwide. However, many languages remain underrepresented due to limited resources. In this paper, we intro…
Speech Codec Probing from Semantic and Phonetic Perspectives
Xuan Shi, Chang Zeng, Tiantian Feng +3
Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems. Speech tokenizers are expected to preserve both semantic and acoustic i…
Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech
Thanapat Trachu, Thanathai Lertpetchpun, Sai Praneeth Karimireddy +1
Recent zero-shot Text-to-Speech (TTS) systems can clone previously unseen voices from only a few seconds of audio. We formulate Speech Generation Speaker Poisoning (SGSP), a task t…
Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis
Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu +4
Many spoken languages, including English, exhibit wide variation in dialects and accents, making accent control an important capability for flexible text-to-speech (TTS) models. Cu…
VoxCog: Towards End-to-End Multilingual Cognitive Impairment Classification through Dialectal Knowledge
Tiantian Feng, Anfeng Xu, Jinkook Lee +1
In this work, we present a novel perspective on cognitive impairment classification from speech by integrating speech foundation models that explicitly recognize speech dialects. O…
End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions
Anfeng Xu, Tiantian Feng, Somer Bishop +2
Accurate transcription and speaker role diarization of child-adult spoken interactions are crucial for developmental and clinical research. However, manual annotation is time-consu…
Assessing Visual Privacy Risks in Multimodal AI: A Novel Taxonomy-Grounded Evaluation of Vision-Language Models
Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +2
Artificial Intelligence have profoundly transformed the technological landscape in recent years. Large Language Models (LLMs) have demonstrated impressive abilities in reasoning, t…
VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks
Efthymios Tsaprazlis, Thanathai Lertpetchpun, Tiantian Feng +2
Voice anonymization aims to conceal speaker identity and attributes while preserving intelligibility, but current evaluations rely almost exclusively on Equal Error Rate (EER) that…
Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe
Tiantian Feng, Kevin Huang, Anfeng Xu +6
We present Voxlect, a novel benchmark for modeling dialects and regional languages worldwide using speech foundation models. Specifically, we report comprehensive benchmark evaluat…
TILES-2018 Sleep Benchmark Dataset: A Longitudinal Wearable Sleep Data Set of Hospital Workers for Modeling and Understanding Sleep Behaviors
Tiantian Feng, Brandon M Booth, Karel Mundnich +4
Sleep is important for everyday functioning, overall well-being, and quality of life. Recent advances in wearable sensing technology have enabled continuous, noninvasive, and cost-…
How to Retrieve Examples in In-context Learning to Improve Conversational Emotion Recognition using Large Language Models?
Mengqi Wang, Tiantian Feng, Shrikanth Narayanan
Large language models (LLMs) have enabled a wide variety of real-world applications in various domains. However, creating a high-performing application with high accuracy remains c…
Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction
Thanathai Lertpetchpun, Tiantian Feng, Dani Byrd +1
Speech emotion recognition (SER) in naturalistic conditions presents a significant challenge for the speech processing community. Challenges include disagreement in labeling among…
Joint ASR and Speaker Role Tagging with Serialized Output Training
Anfeng Xu, Tiantian Feng, Shrikanth Narayanan
Automatic Speech Recognition systems have made significant progress with large-scale pre-trained models. However, most current systems focus solely on transcribing the speech witho…
Developing a Top-tier Framework in Naturalistic Conditions Challenge for Categorized Emotion Prediction: From Speech Foundation Models and Learning Objective to Data Augmentation and Engineering Choices
Tiantian Feng, Thanathai Lertpetchpun, Dani Byrd +1
Speech emotion recognition (SER), particularly for naturally expressed emotions, remains a challenging computational task. Key challenges include the inherent subjectivity in emoti…
CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech
Helin Wang, Jiarui Hai, Dading Chong +11
Recent advancements in generative artificial intelligence have significantly transformed the field of style-captioned text-to-speech synthesis (CapTTS). However, adapting CapTTS to…