GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations

Abstract

Modern deep transfer learning approaches have mainly focused on learning generic feature vectors from one task that are transferable to other tasks, such as word embeddings in language and pretrained convolutional features in vision. However, these approaches usually transfer unary features and largely ignore more structured graphical representations. This work explores the possibility of learning generic latent relational graphs that capture dependencies between pairs of data units (e.g., words or pixels) from large-scale unlabeled data and transferring the graphs to downstream tasks. Our proposed transfer learning framework improves performance on various tasks including question answering, natural language inference, sentiment analysis, and image classification. We also show that the learned graphs are generic enough to be transferred to different embeddings on which the graphs have not been trained (including GloVe embeddings, ELMo embeddings, and task-specific RNN hidden unit), or embedding-free units such as image pixels.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Neural Architecture Search with Reinforcement Learning
• Self-Attention Generative Adversarial Networks
• Pixel Recurrent Neural Networks
• Enhanced LSTM for Natural Language Inference
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Stochastic Answer Networks for Natural Language Inference
• Attending to All Mention Pairs for Full Abstract Biological Relation Extraction

Cited by in corpus (9)

• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Learning to schedule job-shop problems: Representation and policy learning using graph neural network and reinforcement learning
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
• Non-local Recurrent Neural Memory for Supervised Sequence Modeling
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Multilevel Text Alignment with Cross-Document Attention
• Two-Headed Monster And Crossed Co-Attention Networks
• SpaMHMM: Sparse Mixture of Hidden Markov Models for Graph Connected Entities