Time-aware Large Kernel Convolutions

Abstract

To date, most state-of-the-art sequence modeling architectures use attention to build generative models for language based tasks. Some of these models use all the available sequence tokens to generate an attention distribution which results in time complexity of $O(n^2)$. Alternatively, they utilize depthwise convolutions with softmax normalized kernels of size $k$ acting as a limited-window self-attention, resulting in time complexity of $O(k\cdot n)$. In this paper, we introduce Time-aware Large Kernel (TaLK) Convolutions, a novel adaptive convolution operation that learns to predict the size of a summation kernel instead of using a fixed-sized kernel matrix. This method yields a time complexity of $O(n)$, effectively making the sequence encoding process linear to the number of tokens. We evaluate the proposed method on large-scale standard machine translation, abstractive summarization and language modeling datasets and show that TaLK Convolutions constitute an efficient improvement over other attention/convolution based approaches.

arXiv commentAccepted by ICML 2020

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• Convolutional Sequence to Sequence Learning
• Mixed Precision Training
• Reformer: The Efficient Transformer
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Neural Machine Translation in Linear Time
• An Analysis of Neural Language Modeling at Multiple Scales
• Improving Neural Language Models with a Continuous Cache
• Latent Alignment and Variational Attention
• Massively Multilingual Neural Machine Translation
• Accelerating Large-Kernel Convolution Using Summed-Area Tables

Cited by in corpus (1)

• On the Importance of Local Information in Transformer Based Models