Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization

Abstract

Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.

NIPS 2017 camera ready

References in corpus (12)

• Auto-Encoding Variational Bayes
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Wide Residual Networks
• VQA: Visual Question Answering
• Learning Deconvolution Network for Semantic Segmentation
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Deep Networks with Stochastic Depth
• Variational inference for Monte Carlo objectives
• To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout
• Dropout with Expectation-linear Regularization
• Improved Dropout for Shallow and Deep Learning

Cited by in corpus (10)

• Real-time System Identification Using Deep Learning for Linear Processes with Application to Unmanned Aerial Vehicles
• Universal Multimodal Representation for Language Understanding
• Sharing Matters for Generalization in Deep Metric Learning
• Situation-based memory in spiking neuron-astrocyte network
• The stochastic digital human is now enrolling for in silico imaging trials -- Methods and tools for generating digital cohorts
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• A Unified Deep Semantic Expansion Framework for Domain-Generalized Person Re-identification
• Oversampling Log Messages Using a Sequence Generative Adversarial Network for Anomaly Detection and Classification
• CoSeNet: A Novel Approach for Optimal Segmentation of Correlation Matrices
• Context Normalization Layer with Applications