Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions

Murali Karthick Baskar, Andrew Rosenberg, Bhuvana Ramabhadran +2

In this paper, we focus on addressing the constraints faced when applying LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply speech as a prefix to LLMs fo…