Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

Xugang Lu, Peng Shen, Yu Tsao +1

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary…