FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
arXiv:2110.08059
Abstract
When designing Convolutional Neural Networks (CNNs), one must select the size\break of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.
arXiv commentFirst two authors contributed equally to this work
References in corpus (28)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Neural Architecture Search with Reinforcement Learning
- Wide Residual Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- Alias-Free Generative Adversarial Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
- Making Convolutional Networks Shift-Invariant Again
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Efficient-CapsNet: Capsule Network with Self-Attention Routing
- SpinalNet: Deep Neural Network with Gradual Input
- The UEA multivariate time series classification archive, 2018
- Trellis Networks for Sequence Modeling
- Learning Longer-term Dependencies in RNNs with Auxiliary Losses
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
- Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
- Lipschitz Recurrent Neural Networks
- Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
- Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
- CKConv: Continuous Kernel Convolution For Sequential Data
- Parallelizing Legendre Memory Unit Training
- Extended Batch Normalization
- UnICORNN: A recurrent model for learning very long time dependencies
- Deep Continuous Networks