Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques

Abstract

This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. Such models are inspired by the observation that when children pick up a language, they rely on a wide range of indirect and noisy clues, crucially including signals from the visual modality co-occurring with spoken utterances. Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and Cognitive Science. The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work. We then summarize the main modeling architectures and offer an exhaustive overview of the evaluation metrics and analysis techniques.

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Neural Discrete Representation Learning
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Representations of language in a model of visually grounded speech signal
• Language learning using Speech to Image retrieval
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
• SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
• Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
• Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation
• Fast-Slow Transformer for Visually Grounding Speech
• Symbolic inductive bias for visually grounded learning of spoken language
• Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
• Semantic speech retrieval with a visually grounded model of untranscribed speech
• Towards localisation of keywords in speech using weak supervision

Cited by in corpus (10)

• Self-Supervised Speech Representation Learning: A Review
• ConceptBeam: Concept Driven Target Speech Extraction
• Learning English with Peppa Pig
• ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
• Keyword localisation in untranscribed speech using visually grounded speech models
• Video-Guided Curriculum Learning for Spoken Video Grounding
• Wave to Syntax: Probing spoken language models for syntax
• Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning System
• Discrete representations in neural models of spoken language
• Cascaded Multilingual Audio-Visual Learning from Videos