A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization

Abstract

Deep neural networks have introduced novel and useful tools to the machine learning community. Other types of classifiers can potentially make use of these tools as well to improve their performance and generality. This paper reviews the current state of the art for deep learning classifier technologies that are being used outside of deep neural networks. Non-network classifiers can employ many components found in deep neural network architectures. In this paper, we review the feature learning, optimization, and regularization methods that form a core of deep network technologies. We then survey non-neural network learning algorithms that make innovative use of these methods to improve classification. Because many opportunities and challenges still exist, we discuss directions that can be pursued to expand the area of deep learning for a variety of classification algorithms.

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• A Tutorial on Principal Component Analysis
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Deep Learning for Sensor-based Activity Recognition: A Survey
• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Exploring the Limits of Language Modeling
• Data Augmentation Generative Adversarial Networks
• Deep Learning using Linear Support Vector Machines
• AutoAugment: Learning Augmentation Policies from Data
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Fractional Max-Pooling
• Distilling a Neural Network Into a Soft Decision Tree
• A Neural Network Architecture Combining Gated Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection in Network Traffic Data
• All you need is a good init
• On weight initialization in deep neural networks
• Bayesian Recurrent Neural Networks
• Artificial Neural Networks Applied to Taxi Destination Prediction
• Deep Kalman Filters
• Deep Neural Decision Trees
• Avoiding pathologies in very deep networks
• Analyzing noise in autoencoders and deep networks
• Variational Auto-encoded Deep Gaussian Processes
• Recurrent Gaussian Processes
• Deep Robust Kalman Filter
• The Boundary Forest Algorithm for Online Supervised and Unsupervised Learning
• Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees
• Generative Adversarial Forests for Better Conditioned Adversarial Learning