Meta-Embeddings Based On Self-Attention

Abstract

Creating meta-embeddings for better performance in language modelling has received attention lately, and methods based on concatenation or merely calculating the arithmetic mean of more than one separately trained embeddings to perform meta-embeddings have shown to be beneficial. In this paper, we devise a new meta-embedding model based on the self-attention mechanism, namely the Duo. With less than 0.4M parameters, the Duo mechanism achieves state-of-the-art accuracy in text classification tasks such as 20NG. Additionally, we propose a new meta-embedding sequece-to-sequence model for machine translation, which to the best of our knowledge, is the first machine translation model based on more than one word-embedding. Furthermore, it has turned out that our model outperform the Transformer not only in terms of achieving a better result, but also a faster convergence on recognized benchmarks, such as the WMT 2014 English-to-French translation task.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Distributed Representations of Sentences and Documents
• Spectral Networks and Locally Connected Networks on Graphs
• Convolutional Sequence to Sequence Learning
• Convolutional Neural Networks for Sentence Classification
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Generating Long Sequences with Sparse Transformers
• Weighted Transformer Network for Machine Translation
• Massive Exploration of Neural Machine Translation Architectures
• Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
• Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
• An Exploration of Word Embedding Initialization in Deep-Learning Tasks