Depthwise Separable Convolutions for Neural Machine Translation

Abstract

Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results.

References in corpus (8)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Xception: Deep Learning with Depthwise Separable Convolutions
• Neural Machine Translation in Linear Time
• Factorization tricks for LSTM networks
• Encoding Source Language with Convolutional Neural Network for Machine Translation

Cited by in corpus (40)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Non-Autoregressive Neural Machine Translation
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• One Model To Learn Them All
• Fast Decoding in Sequence Models using Discrete Latent Variables
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Understanding Back-Translation at Scale
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Are Transformers universal approximators of sequence-to-sequence functions?
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Neural Architecture Search For Keyword Spotting
• Container: Context Aggregation Network
• Deep Learning Based Chatbot Models
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• Depthwise Convolution is All You Need for Learning Multiple Visual Domains
• Machine Reading Comprehension: a Literature Review
• LightSleepNet: Design of a Personalized Portable Sleep Staging System Based on Single-Channel EEG
• Spatio-Temporal FAST 3D Convolutions for Human Action Recognition
• BeamLearning: an end-to-end Deep Learning approach for the angular localization of sound sources using raw multichannel acoustic pressure data
• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition
• Tensor2Tensor for Neural Machine Translation
• Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
• SmartSplit: Latency-Energy-Memory Optimisation for CNN Splitting on Smartphone Environment
• Scalable Transformers for Neural Machine Translation
• Neural Machine Translation with Byte-Level Subwords
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• CNN Is All You Need
• Better Sign Language Translation with STMC-Transformer
• MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection
• FastFusionNet: New State-of-the-Art for DAWNBench SQuAD
• Dense Information Flow for Neural Machine Translation
• Security Vulnerability Detection Using Deep Learning Natural Language Processing
• Improving Robustness using Joint Attention Network For Detecting Retinal Degeneration From Optical Coherence Tomography Images
• Learning behavioral context recognition with multi-stream temporal convolutional networks
• Neural Machine Translation: A Review and Survey
• Latency-Memory Optimized Splitting of Convolution Neural Networks for Resource Constrained Edge Devices
• Group Equivariant Conditional Neural Processes
• Hierarchical Memory Decoding for Video Captioning
• An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis