WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

Lukas Wolf, Greta Tuckute, Klemen Kotar +4

Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of the…