Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
arXiv:1911.09602
Abstract
In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3\% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.
Camera-ready version for ICLR
References in corpus (10)
- Neural Discrete Representation Learning
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
- Representations of language in a model of visually grounded speech signal
- Generating Diverse High-Fidelity Images with VQ-VAE-2
- A segmental framework for fully-unsupervised large-vocabulary speech recognition
- Unsupervised word segmentation and lexicon discovery using acoustic word embeddings
- Language learning using Speech to Image retrieval
- Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
- Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling
Cited by in corpus (4)
- Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
- ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
- Towards Semi-Supervised Semantics Understanding from Speech
- fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit