Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Abstract

We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.

Patent disclosure, UCSD Docket No. SD2015-184, "Forest Convolutional Neural Network", filed on March 4, 2015. UCSD Docket No. SD2016-053, "Generalizing Pooling Functions in Convolutional Neural Network", filed on Sept 23, 2015

References in corpus (6)

• Going Deeper with Convolutions
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Fractional Max-Pooling
• Deep Networks with Internal Selective Attention through Feedback Connections
• Autoencoder Trees

Cited by in corpus (18)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Temporal Ensembling for Semi-Supervised Learning
• Systematic evaluation of CNN advances on the ImageNet
• Deep Networks with Stochastic Depth
• Deep Residual Networks with Exponential Linear Unit
• What Do We Understand About Convolutional Networks?
• Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• Oriented Response Networks
• Convolutional Residual Memory Networks
• Basic Level Categorization Facilitates Visual Object Recognition
• Knowledge-based Radiation Treatment Planning: A Data-driven Method Survey
• Computational Eco-Systems for Handwritten Digits Recognition
• Improving training of deep neural networks via Singular Value Bounding
• GradNets: Dynamic Interpolation Between Neural Architectures
• GM-Net: Learning Features with More Efficiency
• S3Pool: Pooling with Stochastic Spatial Sampling
• Convolution in Convolution for Network in Network