Language learning using Speech to Image retrieval

Abstract

Humans learn language by interaction with their environment and listening to other humans. It should also be possible for computational models to learn language directly from speech but so far most approaches require text. We improve on existing neural network approaches to create visually grounded embeddings for spoken utterances. Using a combination of a multi-layer GRU, importance sampling, cyclic learning rates, ensembling and vectorial self-attention our results show a remarkable increase in image-caption retrieval performance over previous work. Furthermore, we investigate which layers in the model learn to recognise words in the input. We find that deeper network layers are better at encoding word presence, although the final layer has slightly lower performance. This shows that our visually grounded sentence encoder learns to recognise words from the input even though it is not explicitly trained for word recognition.

arXiv commentSubmitted to InterSpeech 2019

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Skip-Thought Vectors
• Representations of language in a model of visually grounded speech signal
• Snapshot Ensembles: Train 1, get M for free
• Learning semantic sentence representations from visually grounded language without lexical knowledge

Cited by in corpus (17)

• Self-Supervised Speech Representation Learning: A Review
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• Speech-Image Semantic Alignment Does Not Depend on Any Prior Classification Tasks
• improving partition-block-based acoustic echo canceler in under-modeling scenarios
• ConceptBeam: Concept Driven Target Speech Extraction
• Learning English with Peppa Pig
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
• Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
• Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models
• Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech
• S2IGAN: Speech-to-Image Generation via Adversarial Learning
• Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset
• Cascaded Multilingual Audio-Visual Learning from Videos
• Discrete representations in neural models of spoken language
• Semantic sentence similarity: size does not always matter