Learning English with Peppa Pig

Abstract

Recent computational models of the acquisition of spoken language via grounding in perception exploit associations between the spoken and visual modalities and learn to represent speech and visual data in a joint vector space. A major unresolved issue from the point of ecological validity is the training data, typically consisting of images or videos paired with spoken descriptions of what is depicted. Such a setup guarantees an unrealistically strong correlation between speech and the visual data. In the real world the coupling between the linguistic and the visual modality is loose, and often confounded by correlations with non-semantic aspects of the speech signal. Here we address this shortcoming by using a dataset based on the children's cartoon Peppa Pig. We train a simple bi-modal architecture on the portion of the data consisting of dialog between characters, and evaluate on segments containing descriptive narrations. Despite the weak and confounded signal in this training data our model succeeds at learning aspects of the visual semantics of spoken language.

arXiv commentAccepted to TACL

References in corpus (5)

• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Representations of language in a model of visually grounded speech signal
• Language learning using Speech to Image retrieval
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Encoding of phonology in a recurrent neural model of grounded speech