35 citations · 59 across the 25 of their papers we have counts for
28 papers
Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization
Bebe Cosgrove, Aaron Isidore Grace, Weiran Wang
Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucina…
Learning the Robustness Mechanism with Bilevel Optimization
Yiyang Shen, Qihang Lin, Weiran Wang
We propose a distributionally robust learning framework where parameters defining the robustness mechanism are learned from held-out data instead of extensively tuned. Using bileve…
Structured Latent Modeling for Supervised Multimodal Information Decomposition
Wanting Huang, Sanvesh Srivastava, Weiran Wang
Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerg…
Catalogue of Solar Radio Bursts Detected by the LOFAR LV614 Station at the Irbene Observatory during Solar Cycle 25
Jānis Šteinbergs, Dmitrii Kolotkov, Valery M. Nakariakov +3
A catalogue (https://doi.org/10.5281/zenodo.20157853) of solar radio bursts detected with the LV614 LOFAR station at the Irbene Observatory, Latvia, in the stand-alone mode from Ju…
Adaptive Perturbation Selection for Contrastive Audio Decoding
Aaron Isidore Grace, Zhouyuan Huo, Weiran Wang
Large audio-language models (LALMs) frequently hallucinate by overriding acoustic evidence with language priors. While contrastive decoding (CD) offers training-free mitigation, ex…
Penalty-Based First-Order Methods for Bilevel Optimization with Minimax and Constrained Lower-Level Problems
Yiyang Shen, Yutian He, Weiran Wang +1
We study a class of bilevel optimization problems in which both the upper- and lower-level problems have minimax structures. This setting captures a broad range of emerging applica…
Align-Consistency: Improving Non-autoregressive and Semi-supervised ASR with Consistency Regularization
Wanting Huang, Weiran Wang
Consistency regularization (CR) improves the robustness and accuracy of Connectionist Temporal Classification (CTC) by ensuring predictions remain stable across input perturbations…
Disentanglement of Variations with Multimodal Generative Modeling
Yijie Zhang, Yiyang Shen, Weiran Wang
Multimodal data are prevalent across various domains, and learning robust representations of such data is paramount to enhancing generation quality and downstream task performance.…
No Data Wasted: A Semi-supervised Generative Model for Incomplete Multi-view Data Integration with Missing Labels
Yiyang Shen, Weiran Wang
Multi-view learning is widely applied to real-life datasets, but it often suffers from both missing views and missing labels. Prior probabilistic approaches addressed the missing v…
Text Injection for Neural Contextual Biasing
Zhong Meng, Zelin Wu, Rohit Prabhavalkar +5
Neural contextual biasing effectively improves automatic speech recognition (ASR) for crucial phrases within a speaker's context, particularly those that are infrequent in the trai…
Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR
Zelin Wu, Gan Song, Christopher Li +9
Contextual biasing enables speech recognizers to transcribe important phrases in the speaker's context, such as contact names, even if they are rare in, or absent from, the trainin…
Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models
Rohit Prabhavalkar, Zhong Meng, Weiran Wang +7
The accuracy of end-to-end (E2E) automatic speech recognition (ASR) models continues to improve as they are scaled to larger sizes, with some now reaching billions of parameters. W…
USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models
Shaojin Ding, David Qiu, David Rim +10
End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploy…
Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithm
Weiran Wang, Zelin Wu, Diamantino Caseiro +10
Contextual biasing refers to the problem of biasing the automatic speech recognition (ASR) systems towards rare entities that are relevant to the specific user or application scena…
Massive End-to-end Models for Short Search Queries
Weiran Wang, Rohit Prabhavalkar, Dongseong Hwang +11
In this work, we investigate two popular end-to-end automatic speech recognition (ASR) models, namely Connectionist Temporal Classification (CTC) and RNN-Transducer (RNN-T), for of…
Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network
Yiling Huang, Weiran Wang, Guanlong Zhao +3
While standard speaker diarization attempts to answer the question "who spoken when", most of relevant applications in reality are more interested in determining "who spoken what".…
Augmenting conformers with structured state-space sequence models for online speech recognition
Haozhe Shan, Albert Gu, Zhong Meng +3
Online speech recognition, where the model only accesses context to the left, is an important and challenging use case for ASR systems. In this work, we investigate augmenting neur…
Text Injection for Capitalization and Turn-Taking Prediction in Speech Models
Shaan Bijwadia, Shuo-yiin Chang, Weiran Wang +3
Text injection for automatic speech recognition (ASR), wherein unpaired text-only data is used to supplement paired audio-text data, has shown promising improvements for word error…
Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR
Rami Botros, Anmol Gulati, Tara N. Sainath +5
Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers. With limited memory bandwidth, reading these from…
JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition
Zhong Meng, Weiran Wang, Rohit Prabhavalkar +7
We propose JEIT, a joint end-to-end (E2E) model and internal language model (ILM) training method to inject large-scale unpaired text into ILM during E2E training which improves ra…
JOIST: A Joint Speech and Text Streaming Model For ASR
Tara N. Sainath, Rohit Prabhavalkar, Ankur Bapna +6
We present JOIST, an algorithm to train a streaming, cascaded, encoder end-to-end (E2E) model with both speech-text paired inputs, and text-only unpaired inputs. Unlike previous wo…
Improving Deliberation by Text-Only and Semi-Supervised Training
Ke Hu, Tara N. Sainath, Yanzhang He +4
Text-only and semi-supervised training based on audio-only data has gained popularity recently due to the wide availability of unlabeled text and speech data. In this work, we prop…
Streaming Align-Refine for Non-autoregressive Deliberation
Weiran Wang, Ke Hu, Tara N. Sainath
We propose a streaming non-autoregressive (non-AR) decoding algorithm to deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm facilitates a simple greedy d…
Improving Rare Word Recognition with LM-aware MWER Training
Weiran Wang, Tongzhou Chen, Tara N. Sainath +10
Language models (LMs) significantly improve the recognition accuracy of end-to-end (E2E) models on words rarely seen during training, when used in either the shallow fusion or the…