Audio Token Attention Is Predictable Before the Language Model Runs

Kyoungjun Park, Yunzhe Li, Lili Qiu

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, w…