LLM can Read Spectrogram: Encoder-free Speech-Language Modeling

Ruchao Fan, Yiming Wang, Yuxuan Hu +6

Recent speech-aware large language models (Speech-LLMs) rely on pre-trained speech encoders to convert audio into semantic/acoustic rich representations consumable by LLM. In this…