6 citations · 33 across the 44 of their papers we have counts for
47 papers
ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding
Ashutosh Chaubey, Jinxi Guo, Minh Tran +2
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data…
VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation
Yancheng Wang, Osama Hanna, Ruiming Xie +11
Emotion recognition in speech presents a complex multimodal challenge, requiring comprehension of both linguistic content and vocal expressivity, particularly prosodic features suc…
Can Speech LLMs Think while Listening?
Yi-Jen Shih, Desh Raj, Chunyang Wu +6
Recent advances in speech large language models (speech LLMs) have enabled seamless spoken interactions, but these systems still struggle with complex reasoning tasks. Previously,…
Dialect Normalization using Large Language Models and Morphological Rules
Antonios Dimakis, John Pavlopoulos, Antonios Anastasopoulos
Natural language understanding systems struggle with low-resource languages, including many dialects of high-resource ones. Dialect-to-standard normalization attempts to tackle thi…
CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR
Wei Zhou, Junteng Jia, Leda Sari +2
CTC compressor can be an effective approach to integrate audio encoders to decoder-only models, which has gained growing interest for different speech applications. In this work, w…
When Every Token Counts: Optimal Segmentation for Low-Resource Language Models
Bharath Raj, Garvit Suri, Vikrant Dewangan +1
Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model p…
Efficient Streaming LLM for Speech Recognition
Junteng Jia, Gil Keren, Wei Zhou +6
Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently,…
Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech
Wonjune Kang, Junteng Jia, Chunyang Wu +8
This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system w…
M-BEST-RQ: A Multi-Channel Speech Foundation Model for Smart Glasses
Yufeng Yang, Desh Raj, Ju Lin +8
The growing popularity of multi-channel wearable devices, such as smart glasses, has led to a surge of applications such as targeted speech recognition and enhanced hearing. Howeve…
Faster Speech-LLaMA Inference with Multi-token Prediction
Desh Raj, Gil Keren, Junteng Jia +2
Large language models (LLMs) have become proficient at solving a wide variety of tasks, including those involving multi-modal inputs. In particular, instantiating an LLM (such as L…
Towards measuring fairness in speech recognition: Fair-Speech dataset
Irina-Elena Veliche, Zhuangqun Huang, Vineeth Ayyat Kochaniyan +3
The current public datasets for speech recognition (ASR) tend not to focus specifically on the fairness aspect, such as performance across different demographic groups. This paper…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
Token-Weighted RNN-T for Learning from Flawed Data
Gil Keren, Wei Zhou, Ozlem Kalinli
ASR models are commonly trained with the cross-entropy criterion to increase the probability of a target token sequence. While optimizing the probability of all tokens in the targe…
Effective internal language model training and fusion for factorized transducer model
Jinxi Guo, Niko Moritz, Yingyi Ma +6
The internal language model (ILM) of the neural transducer has been widely studied. In most prior work, it is mainly used for estimating the ILM score and is subsequently subtracte…
IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages
Tahir Javed, Janki Atul Nawale, Eldho Ittan George +18
We present INDICVOICES, a dataset of natural and spontaneous speech containing a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio from 16237 speaker…
AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs
Yassir Fathullah, Chunyang Wu, Egor Lakomkin +7
In this work, we extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities while maintaining the wide range of original L…
End-to-End Speech Recognition Contextualization with Large Language Models
Egor Lakomkin, Chunyang Wu, Yassir Fathullah +3
In recent years, Large Language Models (LLMs) have garnered significant attention from the research community due to their exceptional performance and generalization capabilities.…
Augmenting text for spoken language understanding with Large Language Models
Roshan Sharma, Suyoun Kim, Daniel Lazar +7
Spoken semantic parsing (SSP) involves generating machine-comprehensible parses from input speech. Training robust models for existing application domains represented in training d…
Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model
Jiamin Xie, Ke Li, Jinxi Guo +7
Neural network pruning offers an effective method for compressing a multilingual automatic speech recognition (ASR) model with minimal performance loss. However, it entails several…
TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models
Yuan Shangguan, Haichuan Yang, Danni Li +11
Automatic Speech Recognition (ASR) models need to be optimized for specific hardware before they can be deployed on devices. This can be done by tuning the model's hyperparameters…
Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding
Suyoun Kim, Akshat Shrivastava, Duc Le +3
End-to-end (E2E) spoken language understanding (SLU) systems that generate a semantic parse from speech have become more promising recently. This approach uses a single model that…
Prompting Large Language Models with Speech Recognition Abilities
Yassir Fathullah, Chunyang Wu, Egor Lakomkin +9
Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. I…
Towards Selection of Text-to-speech Data to Augment ASR Training
Shuo Liu, Leda Sarı, Chunyang Wu +4
This paper presents a method for selecting appropriate synthetic speech samples from a given large text-to-speech (TTS) dataset as supplementary training data for an automatic spee…
Multi-Head State Space Model for Speech Recognition
Yassir Fathullah, Chunyang Wu, Yuan Shangguan +8
State space models (SSMs) have recently shown promising results on small-scale sequence and language modelling tasks, rivalling and outperforming many attention-based approaches. I…