LF${}^2$AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

Santiago Cuervo, Adel Moumen, Yanis Labrak +5

Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively levera…