From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
arXiv:1610.03342
Abstract
We present a model of visually-grounded language learning based on stacked gated recurrent neural networks which learns to predict visual features given an image description in the form of a sequence of phonemes. The learning task resembles that faced by human language learners who need to discover both structure and meaning from noisy and ambiguous data across modalities. We show that our model indeed learns to predict features of the visual context given phonetically transcribed image descriptions, and show that it represents linguistic information in a hierarchy of levels: lower layers in the stack are comparatively more sensitive to form, whereas higher layers are more sensitive to meaning.
arXiv commentAccepted at COLING 2016
References in corpus (9)
- Scikit-learn: Machine Learning in Python
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Deep Residual Learning for Image Recognition
- Theano: new features and speed improvements
- Visualizing and Understanding Recurrent Networks
- Skip-Thought Vectors
- Neural Language Correction with Character-Based Attention
- Text segmentation with character-level text embeddings
- Representation of linguistic form and function in recurrent neural networks
Cited by in corpus (10)
- What do Neural Machine Translation Models Learn about Morphology?
- Representations of language in a model of visually grounded speech signal
- Imagination improves Multimodal Translation
- The emergence of number and syntax units in LSTM language models
- Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
- Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
- Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
- Learning to Discover, Ground and Use Words with Segmental Neural Language Models
- Towards localisation of keywords in speech using weak supervision
- Analysis Methods in Neural Language Processing: A Survey