Two-Headed Monster And Crossed Co-Attention Networks
arXiv:1911.03897
Abstract
This paper presents some preliminary investigations of a new co-attention mechanism in neural transduction models. We propose a paradigm, termed Two-Headed Monster (THM), which consists of two symmetric encoder modules and one decoder module connected with co-attention. As a specific and concrete implementation of THM, Crossed Co-Attention Networks (CCNs) are designed based on the Transformer model. We demonstrate CCNs on WMT 2014 EN-DE and WMT 2016 EN-FI translation tasks and our model outperforms the strong Transformer baseline by 0.51 (big) and 0.74 (base) BLEU points on EN-DE and by 0.17 (big) and 0.47 (base) BLEU points on EN-FI.
References in corpus (10)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Teaching Machines to Read and Comprehend
- Attention-Based Models for Speech Recognition
- Reasoning about Entailment with Neural Attention
- Multilingual Denoising Pre-training for Neural Machine Translation
- Attentive Pooling Networks
- Pay Less Attention with Lightweight and Dynamic Convolutions
- The Evolved Transformer
- Incorporating BERT into Neural Machine Translation
- GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations