Learning Implicitly Recurrent CNNs Through Parameter Sharing

Abstract

We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Compared to traditional CNNs, we demonstrate substantial parameter savings on standard image classification tasks, while maintaining accuracy. Our simple parameter sharing scheme, though defined via soft weights, in practice often yields trained networks with near strict recurrent structure; with negligible side effects, they convert into networks with actual loops. Training these networks thus implicitly involves discovery of suitable recurrent architectures. Though considering only the design aspect of recurrent links, our trained networks achieve accuracy competitive with those built using state-of-the-art neural architecture search (NAS) procedures. Our hybridization of recurrent and convolutional networks may also represent a beneficial architectural bias. Specifically, on synthetic tasks which are algorithmic in nature, our hybrid networks both train faster and extrapolate better to test examples outside the span of the training set.

Published as a conference paper at ICLR 2019

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Improved Regularization of Convolutional Neural Networks with Cutout
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Efficient Neural Architecture Search via Parameter Sharing
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Residual Connections Encourage Iterative Inference

Cited by in corpus (13)

• Discovering Neural Wirings
• Winning the Lottery with Continuous Sparsification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Parameterized Hypercomplex Graph Neural Networks for Graph Classification
• Relaxed Parameter Sharing: Effectively Modeling Time-Varying Relationships in Clinical Time-Series
• Growing Efficient Deep Networks by Structured Continuous Sparsification
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• Restoring Spatially-Heterogeneous Distortions using Mixture of Experts Network
• Neural Parameter Allocation Search
• Full-Stack Filters to Build Minimum Viable CNNs
• A Survey on Green Deep Learning
• Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks