10 papers
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Å imon SedláÄek, Sara Barahona, Bolaji Yusuf +9
Reliable assessment of the abilities of large audio language models (LALMs) is essential to advancing the state of the art. As benchmarks rapidly evolve to incorporate complex reas…
Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking
Katia Vendrame, Bolaji Yusuf, Santosh Kesiraju +3
End-to-end spoken dialogue state tracking (DST) is made difficult by the tandem of having to handle speech input and data scarcity. Combining speech foundation encoders and large l…
FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings
Santosh Kesiraju, Bolaji Yusuf, Å imon SedláÄek +2
This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from mult…
DeCRED: Decoder-Centric Regularization for Encoder-Decoder Based Speech Recognition
Alexander Polok, Santosh Kesiraju, Karel Beneš +3
This paper presents a simple yet effective regularization for the internal language model induced by the decoder in encoder-decoder ASR models, thereby improving robustness and gen…
Factors affecting the in-context learning abilities of LLMs for dialogue state tracking
Pradyoth Hegde, Santosh Kesiraju, Jan Å vec +5
This study explores the application of in-context learning (ICL) to the dialogue state tracking (DST) problem and investigates the factors that influence its effectiveness. We use…
Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
Å imon SedláÄek, Bolaji Yusuf, Ján Å vec +4
In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully…