SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data

Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang +5

The recently proposed visually grounded speech model SpeechCLIP is an innovative framework that bridges speech and text through images via CLIP without relying on text transcriptio…