Depthwise Separable Convolutions for Neural Machine Translation

Abstract

Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results.

References in corpus (4)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Neural Machine Translation in Linear Time
• Encoding Source Language with Convolutional Neural Network for Machine Translation

Cited by in corpus (10)

• One Model To Learn Them All
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Are Transformers universal approximators of sequence-to-sequence functions?
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Depthwise Convolution is All You Need for Learning Multiple Visual Domains
• Machine Reading Comprehension: a Literature Review
• CNN Is All You Need
• FastFusionNet: New State-of-the-Art for DAWNBench SQuAD
• Hierarchical Memory Decoding for Video Captioning