Batch-normalized Maxout Network in Network

Abstract

This paper reports a novel deep architecture referred to as Maxout network In Network (MIN), which can enhance model discriminability and facilitate the process of information abstraction within the receptive field. The proposed network adopts the framework of the recently developed Network In Network structure, which slides a universal approximator, multilayer perceptron (MLP) with rectifier units, to exact features. Instead of MLP, we employ maxout MLP to learn a variety of piecewise linear activation functions and to mediate the problem of vanishing gradients that can occur when using rectifier units. Moreover, batch normalization is applied to reduce the saturation of maxout units by pre-conditioning the model and dropout is applied to prevent overfitting. Finally, average pooling is used in all pooling layers to regularize maxout MLP in order to facilitate information abstraction in every receptive field while tolerating the change of object position. Because average pooling preserves all features in the local patch, the proposed MIN model can enforce the suppression of irrelevant information during training. Our experiments demonstrated the state-of-the-art classification performance when the MIN model was applied to MNIST, CIFAR-10, and CIFAR-100 datasets and comparable performance for SVHN dataset.

References in corpus (2)

• Improving neural networks by preventing co-adaptation of feature detectors
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Cited by in corpus (19)

• A survey on modern trainable activation functions
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Convolutional Neural Fabrics
• Dynamic Routing Between Capsules
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST)
• Striking the Right Balance with Uncertainty
• A Broad-Spectrum Diffractive Network via Ensemble Learning
• Enhancing approximation abilities of neural networks by training derivatives
• Max-margin Class Imbalanced Learning with Gaussian Affinity
• Constructing Deep Neural Networks by Bayesian Network Structure Learning
• CIFAR-10: KNN-based Ensemble of Classifiers
• Computational Eco-Systems for Handwritten Digits Recognition
• Predictive Uncertainty through Quantization
• Single Image Super-Resolution Using Lightweight CNN with Maxout Units
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Deep Competitive Pathway Networks
• Unbounded Output Networks for Classification
• Neighborhood Watch: Representation Learning with Local-Margin Triplet Loss and Sampling Strategy for K-Nearest-Neighbor Image Classification