WeightNet: Revisiting the Design Space of Weight Networks

Abstract

We present a conceptually simple, flexible and effective framework for weight generating networks. Our approach is general that unifies two current distinct and extremely effective SENet and CondConv into the same framework on weight space. The method, called WeightNet, generalizes the two methods by simply adding one more grouped fully-connected layer to the attention activation layer. We use the WeightNet, composed entirely of (grouped) fully-connected layers, to directly output the convolutional weight. WeightNet is easy and memory-conserving to train, on the kernel space instead of the feature space. Because of the flexibility, our method outperforms existing approaches on both ImageNet and COCO detection tasks, achieving better Accuracy-FLOPs and Accuracy-Parameter trade-offs. The framework on the flexible weight space has the potential to further improve the performance. Code is available at https://github.com/megvii-model/WeightNet.

arXiv commentECCV 2020

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Squeeze-and-Excitation Networks
• Dynamic Filter Networks
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• Slimmable Neural Networks
• HyperNetworks
• Contextual Parameter Generation for Universal Neural Machine Translation
• SplineNets: Continuous Neural Decision Graphs
• You Look Twice: GaterNet for Dynamic Filter Selection in CNNs

Cited by in corpus (2)

• Revisiting Dynamic Convolution via Matrix Decomposition
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs