Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution

Abstract

In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies, and design a novel Octave Convolution (OctConv) operation to store and process feature maps that vary spatially "slower" at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale methods, OctConv is formulated as a single, generic, plug-and-play convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing convolutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9% top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs.

arXiv commentAccepted to ICCV 2019

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• A Short Note about Kinetics-600
• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
• SlowFast Networks for Video Recognition
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
• Dual Path Networks
• Graph-Based Global Reasoning Networks
• ELASTIC: Improving CNNs with Dynamic Scaling Policies

Cited by in corpus (40)

• Res2Net: A New Multi-scale Backbone Architecture
• Is Space-Time Attention All You Need for Video Understanding?
• XNOR-Net++: Improved Binary Neural Networks
• Multiscale Vision Transformers
• AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filter
• xQSM: Quantitative Susceptibility Mapping with Octave Convolutional and Noise Regularized Neural Networks
• Accurate Retinal Vessel Segmentation via Octave Convolution Neural Network
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• Provably scale-covariant continuous hierarchical networks based on scale-normalized differential expressions coupled in cascade
• Generalized Octave Convolutions for Learned Multi-Frequency Image Compression
• Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales
• First-Generation Inference Accelerator Deployment at Facebook
• Searching for Winograd-aware Quantized Networks
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• Highly Efficient Salient Object Detection with 100K Parameters
• Highly Efficient Natural Image Matting
• A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets
• Deep Learning Training on the Edge with Low-Precision Posits
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Hybrid Composition with IdleBlock: More Efficient Networks for Image Recognition
• Spectral Graph Attention Network with Fast Eigen-approximation
• Learned Variable-Rate Multi-Frequency Image Compression using Modulated Generalized Octave Convolution
• MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
• FAN: Frequency Aggregation Network for Real Image Super-resolution
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Split to Be Slim: An Overlooked Redundancy in Vanilla Convolution
• Semi-supervised Learning for Few-shot Image-to-Image Translation
• ResNetX: a more disordered and deeper network architecture
• Direction-aware Feature-level Frequency Decomposition for Single Image Deraining
• U2-ONet: A Two-level Nested Octave U-structure with Multiscale Attention Mechanism for Moving Instances Segmentation
• Learning Omni-frequency Region-adaptive Representations for Real Image Super-Resolution
• MSDU-net: A Multi-Scale Dilated U-net for Blur Detection
• DONet: Dual-Octave Network for Fast MR Image Reconstruction
• Dense Pruning of Pointwise Convolutions in the Frequency Domain
• CTM: Collaborative Temporal Modeling for Action Recognition
• Glioblastoma Multiforme Prognosis: MRI Missing Modality Generation, Segmentation and Radiogenomic Survival Prediction
• Efficient Modelling Across Time of Human Actions and Interactions
• Deep grey matter quantitative susceptibility mapping from small spatial coverages using deep learning
• Octave Mix: Data augmentation using frequency decomposition for activity recognition