Mind the Gap: Aligning the Brain with Language Models Requires a Nonlinear and Multimodal Approach

Danny Dongyeop Han, Yunju Cho, Jiook Cha +1

Self-supervised language and audio models effectively predict brain responses to speech. However, traditional prediction models rely on linear mappings from unimodal features, desp…