CKConv: Continuous Kernel Convolution For Sequential Data

Abstract

Conventional neural architectures for sequential data present important limitations. Recurrent networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional networks are unable to handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that all these problems can be solved by formulating convolutional kernels in CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) allows us to model arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a faster and simpler manner.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• On the Number of Linear Regions of Deep Neural Networks
• Searching for Activation Functions
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• How to Construct Deep Recurrent Neural Networks
• Stand-Alone Self-Attention in Vision Models
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• The UEA multivariate time series classification archive, 2018
• Trellis Networks for Sequence Modeling
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Complexity of Linear Regions in Deep Networks
• LieTransformer: Equivariant self-attention for Lie Groups
• Interpolated Convolutional Networks for 3D Point Cloud Understanding
• Generative Models as Distributions of Functions
• Group Equivariant Stand-Alone Self-Attention For Vision

Cited by in corpus (5)

• Efficiently Modeling Long Sequences with Structured State Spaces
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Generative Models as Distributions of Functions
• Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups