Learning language through pictures

Abstract

We propose Imaginet, a model of learning visually grounded representations of language from coupled textual and visual input. The model consists of two Gated Recurrent Unit networks with shared word embeddings, and uses a multi-task objective by receiving a textual description of a scene and trying to concurrently predict its visual representation and the next word in the sentence. Mimicking an important aspect of human language learning, it acquires meaning representations for individual words from descriptions of visual scenes. Moreover, it learns to effectively use sequential structure in semantic interpretation of multi-word phrases.

To appear at ACL 2015

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Theano: new features and speed improvements
• Learning a Recurrent Visual Representation for Image Caption Generation
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• From Captions to Visual Concepts and Back

Cited by in corpus (6)

• Representations of language in a model of visually grounded speech signal
• Imagination improves Multimodal Translation
• Representation of linguistic form and function in recurrent neural networks
• A Computational Model of Early Word Learning from the Infant's Point of View
• Improving Visually Grounded Sentence Representations with Self-Attention
• Word learning and the acquisition of syntactic--semantic overhypotheses