53 citations · 137 across the 20 of their papers we have counts for
21 papers
Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer Workspace
Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar +8
An audio language model is a black box in a specific way: we see what it says, never what it works out on the way there, and chain-of-thought monitoring helps only if the model wri…
Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition
Mengze Hong, Yi Gu, Di Jiang +4
Training automatic speech recognition (ASR) models increasingly relies on decentralized federated learning to ensure data privacy and accessibility, producing multiple local models…
Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards
Jiajun Fan, Roger Ren, Jingyuan Li +6
The role of reasoning in Audio Large Language Models remains widely underexplored, as introducing a reasoning process often degrades rather than improves performance during inferen…
Group Relative Policy Optimization for Speech Recognition
Prashanth Gurunath Shivakumar, Yile Gu, Ankur Gandhe +1
Speech Recognition has seen a dramatic shift towards adopting Large Language Models (LLMs). This shift is partly driven by good scalability properties demonstrated by LLMs, ability…
Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav +4
While textless Spoken Language Models (SLMs) have shown potential in end-to-end speech-to-speech modeling, they still lag behind text-based Large Language Models (LLMs) in terms of…
Speech Recognition Rescoring with Large Speech-Text Foundation Models
Prashanth Gurunath Shivakumar, Jari Kolehmainen, Aditya Gourav +4
Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often…
Multi-Modal Retrieval For Large Language Model Based Speech Recognition
Jari Kolehmainen, Aditya Gourav, Prashanth Gurunath Shivakumar +5
Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important…
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe +6
Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralingui…
Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks
Kevin Everson, Yile Gu, Huck Yang +10
In the realm of spoken language understanding (SLU), numerous natural language understanding (NLU) methodologies have been adapted by supplying large language models (LLMs) with tr…
Discriminative Speech Recognition Rescoring with Pre-trained Language Models
Prashanth Gurunath Shivakumar, Jari Kolehmainen, Yile Gu +3
Second pass rescoring is a critical component of competitive automatic speech recognition (ASR) systems. Large language models have demonstrated their ability in using pre-trained…
Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu +3
We explore the ability of large language models (LLMs) to act as speech recognition post-processors that perform rescoring and error correction. Our first focus is on instruction p…
Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen +15
We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have s…
Personalization for BERT-based Discriminative Speech Recognition Rescoring
Jari Kolehmainen, Yile Gu, Aditya Gourav +4
Recognition of personalized content remains a challenge in end-to-end speech recognition. We explore three novel approaches that use personalized content in a neural rescoring step…
Scaling Laws for Discriminative Speech Recognition Rescoring Models
Yile Gu, Prashanth Gurunath Shivakumar, Jari Kolehmainen +3
Recent studies have found that model performance has a smooth power-law relationship, or scaling laws, with training data and model size, for a wide range of problems. These scalin…
Distillation Strategies for Discriminative Speech Recognition Rescoring
Prashanth Gurunath Shivakumar, Jari Kolehmainen, Yile Gu +3
Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by ex…
RescoreBERT: Discriminative Speech Recognition Rescoring with BERT
Liyan Xu, Yile Gu, Jari Kolehmainen +5
Second-pass rescoring is an important component in automatic speech recognition (ASR) systems that is used to improve the outputs from a first-pass decoder by implementing a lattic…
Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognition
Chao-Han Huck Yang, Zeeshan Ahmed, Yile Gu +5
In this work, we aim to enhance the system robustness of end-to-end automatic speech recognition (ASR) against adversarially-noisy speech examples. We focus on a rigorous and empir…
Domain-aware Neural Language Models for Speech Recognition
Linda Liu, Yile Gu, Aditya Gourav +5
As voice assistants become more ubiquitous, they are increasingly expected to support and perform well on a wide variety of use-cases across different domains. We present a domain-…
Personalization Strategies for End-to-End Speech Recognition Systems
Aditya Gourav, Linda Liu, Ankur Gandhe +9
The recognition of personalized content, such as contact names, remains a challenging problem for end-to-end speech recognition systems. In this work, we demonstrate how first and…
Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion
Vijay Ravi, Yile Gu, Ankur Gandhe +5
End-to-end automatic speech recognition (ASR) systems, such as recurrent neural network transducer (RNN-T), have become popular, but rare word remains a challenge. In this paper, w…
Multi-task Language Modeling for Improving Speech Recognition of Rare Words
Chao-Han Huck Yang, Linda Liu, Ankur Gandhe +4
End-to-end automatic speech recognition (ASR) systems are increasingly popular due to their relative architectural simplicity and competitive performance. However, even though the…