3 citations · 16 across the 14 of their papers we have counts for
15 papers
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Jagadeesh Balam, Travis Bartley, Edresson Casanova +52
We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder an…
A frontend-backend architecture for tool calls in full-duplex speech models
Ke Hu, Slyne Deng, Chen Chen +10
Full-duplex speech-to-speech (S2S) models provide natural, low-latency conversational interaction and would benefit from the ability to use external tools and complete voice-agent…
MLLM-based Speech Recognition: When and How is Multimodality Beneficial?
Yiwen Guan, Viet Anh Trinh, Vivek Voleti +1
Recent advances in multi-modal large language models (MLLMs) have opened new possibilities for unified modeling of speech, text, images, and other modalities. Building on our prior…
Improving Speech Recognition of Named Entities in Classroom Speech with LLM Revision and Phonetic-Semantic Context
Viet Anh Trinh, Xinlu He, Jacob Whitehill
Classroom speech and lectures often contain named entities (NEs) such as names of people and special terminology. While automatic speech recognition (ASR) systems have achieved rem…
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
Yiwen Guan, Viet Anh Trinh, Vivek Voleti +1
Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities imp…
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing
Viet Anh Trinh, Rosy Southwell, Yiwen Guan +3
Recent work on discrete speech tokenization has paved the way for models that can seamlessly perform multiple tasks across modalities, e.g., speech recognition, text to speech, spe…
Two-pass Endpoint Detection for Speech Recognition
Anirudh Raju, Aparna Khare, Di He +9
Endpoint (EP) detection is a key component of far-field speech recognition systems that assist the user through voice commands. The endpoint detector has to trade-off between accur…
Adaptive Endpointing with Deep Contextual Multi-armed Bandits
Do June Min, Andreas Stolcke, Anirudh Raju +4
Current endpointing (EP) solutions learn in a supervised framework, which does not allow the model to incorporate feedback and improve in an online setting. Also, it is a common pr…
Reducing Geographic Disparities in Automatic Speech Recognition via Elastic Weight Consolidation
Viet Anh Trinh, Pegah Ghahremani, Brian King +3
We present an approach to reduce the performance disparity between geographic regions without degrading performance on the overall user population for ASR. A popular approach is to…
ImportantAug: a data augmentation agent for speech
Viet Anh Trinh, Hassan Salami Kavaki, Michael I Mandel
We introduce ImportantAug, a technique to augment training data for speech classification and recognition models by adding noise to unimportant regions of the speech and not to imp…
Unsupervised Speech Enhancement with speech recognition embedding and disentanglement losses
Viet Anh Trinh, Sebastian Braun
Speech enhancement has recently achieved great success with various deep learning methods. However, most conventional speech enhancement systems are trained with supervised methods…
Combining Spatial Clustering with LSTM Speech Models for Multichannel Speech Enhancement
Felix Grezes, Zhaoheng Ni, Viet Anh Trinh +1
Recurrent neural networks using the LSTM architecture can achieve significant single-channel noise reduction. It is not obvious, however, how to apply them to multi-channel inputs…
Improved MVDR Beamforming Using LSTM Speech Models to Clean Spatial Clustering Masks
Zhaoheng Ni, Felix Grezes, Viet Anh Trinh +1
Spatial clustering techniques can achieve significant multi-channel noise reduction across relatively arbitrary microphone configurations, but have difficulty incorporating a detai…
Enhancement of Spatial Clustering-Based Time-Frequency Masks using LSTM Neural Networks
Felix Grezes, Zhaoheng Ni, Viet Anh Trinh +1
Recent works have shown that Deep Recurrent Neural Networks using the LSTM architecture can achieve strong single-channel speech enhancement by estimating time-frequency masks. How…
Large scale evaluation of importance maps in automatic speech recognition
Viet Anh Trinh, Michael I Mandel
In this paper, we propose a metric that we call the structured saliency benchmark (SSBM) to evaluate importance maps computed for automatic speech recognizers on individual utteran…