Dissecting Contextual Word Embeddings: Architecture and Representation

Abstract

Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned. We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks. Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers. Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.

arXiv commentEMNLP 2018

References in corpus (19)

• Efficient Estimation of Word Representations in Vector Space
• ADADELTA: An Adaptive Learning Rate Method
• WaveNet: A Generative Model for Raw Audio
• Axiomatic Attribution for Deep Networks
• Convolutional Sequence to Sequence Learning
• Training Very Deep Networks
• Enhanced LSTM for Natural Language Inference
• Exploring the Limits of Language Modeling
• Visualizing and Understanding Recurrent Networks
• Regularizing and Optimizing LSTM Language Models
• Understanding Neural Networks through Representation Erasure
• What do Neural Machine Translation Models Learn about Morphology?
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Learned in Translation: Contextualized Word Vectors
• Natural Language Inference over Interaction Space
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• Constituency Parsing with a Self-Attentive Encoder
• Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

Cited by in corpus (3)

• Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL
• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
• Grounding Representation Similarity with Statistical Testing