Recurrent Convolution for Compact and Cost-Adjustable Neural Networks: An Empirical Study
arXiv:1902.09809
Abstract
Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple steps, which is originally proposed to model time-space signals. We argue that RC can be viewed as a model compression strategy for deep convolutional neural networks. RC reduces the redundancy across layers. However, the performance of an RC network is not satisfactory if we directly unroll the same kernels multiple steps. We propose a simple yet effective variant which improves the RC networks: the batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps. Moreover, we verify that RC can perform cost-adjustable inference which is achieved by varying its unrolling steps. We learn double independent BN layers for cost-adjustable RC networks, i.e. independent w.r.t both the unrolling steps of current cell and upstream cell. We provide insights on why the proposed method works successfully. Experiments on both image classification and image denoise demonstrate the effectiveness of our method.
arXiv comment8 pages; preprint; work in progress
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Density estimation using Real NVP
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- Towards the Limit of Network Quantization
- Doubly Nested Network for Resource-Efficient Inference