The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoders

Adrian Sauter, Willem Zuidema, Marianne de Heer Kloots

How does visual information included in training affect language processing in audio- and text-based deep learning models? We explore how such visual grounding affects model-intern…