Injecting Hierarchy with U-Net Transformers
arXiv:1910.10488
Abstract
The Transformer architecture has become increasingly popular over the past two years, owing to its impressive performance on a number of natural language processing (NLP) tasks. However, all Transformer computations occur at the level of word representations and therefore, it may be argued that Transformer models do not explicitly attempt to learn hierarchical structure which is widely assumed to be integral to language. In the present work, we introduce hierarchical processing into the Transformer model, taking inspiration from the U-Net architecture, popular in computer vision for its hierarchical view of natural images. We empirically demonstrate that the proposed architecture outperforms both the vanilla Transformer and some strong baselines in the domain of chit-chat dialogue.
arXiv comment10 pages
References in corpus (10)
- The Curious Case of Neural Text Degeneration
- Generative Modeling by Estimating Gradients of the Data Distribution
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Generating Long Sequences with Sparse Transformers
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- Hierarchical Transformers for Multi-Document Summarization
- A Hierarchical Latent Structure for Variational Conversation Modeling