Deep Learning in Neural Networks: An Overview

Abstract

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

88 pages, 888 references

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• On the difficulty of training Recurrent Neural Networks
• Sum-Product Networks: A New Deep Architecture
• No More Pesky Learning Rates
• Simulated Car Racing Championship: Competition Software Manual
• Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
• Self-Delimiting Neural Networks
• The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations
• Variational inference of latent state sequences using Recurrent Networks

Cited by in corpus (132)

• ANI-1: An extensible neural network potential with DFT accuracy at force field computational cost
• Phase recovery and holographic image reconstruction using deep learning in neural networks
• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• Deep Learning Microscopy
• A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community
• Metaheuristic Design of Feedforward Neural Networks: A Review of Two Decades of Research
• Interaction Networks for Learning about Objects, Relations and Physics
• Group Sparse Regularization for Deep Neural Networks
• Deep Learning: A Critical Appraisal
• Discovering Phases, Phase Transitions and Crossovers through Unsupervised Machine Learning: A critical examination
• A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines
• Hierarchical Multiscale Recurrent Neural Networks
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• Machine learning vortices at the Kosterlitz-Thouless transition
• Deep Learning for Time-Series Analysis
• Deep Learning and Its Applications to Machine Health Monitoring: A Survey
• Towards Deep Symbolic Reinforcement Learning
• Deep Multi-scale Location-aware 3D Convolutional Neural Networks for Automated Detection of Lacunes of Presumed Vascular Origin
• Deep Networks with Internal Selective Attention through Feedback Connections
• Deep Learning: A Bayesian Perspective
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Evaluating Two-Stream CNN for Video Classification
• Machine Learning of Calabi-Yau Volumes
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
• Neural Networks for Modeling and Control of Particle Accelerators
• ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI
• On the Origin of Deep Learning
• Deep learning for galaxy surface brightness profile fitting
• Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition
• Power of Deep Learning for Channel Estimation and Signal Detection in OFDM Systems
• Implicit Regularization in Deep Learning
• Extraction of Skin Lesions from Non-Dermoscopic Images Using Deep Learning
• Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations
• Benchmarking State-of-the-Art Deep Learning Software Tools
• A Unified Neural Network Approach for Estimating Travel Time and Distance for a Taxi Trip
• A Probabilistic Theory of Deep Learning
• Training of Deep Neural Networks based on Distance Measures using RMSProp
• Deep learning for plasma tomography using the bolometer system at JET
• Neural Information Retrieval: A Literature Review
• Sea Level Anomaly Prediction using Recurrent Neural Networks
• Individual Recognition in Schizophrenia using Deep Learning Methods with Random Forest and Voting Classifiers: Insights from Resting State EEG Streams
• Local minima in training of neural networks
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Deep Recurrent Generative Decoder for Abstractive Text Summarization
• Counterfactual Prediction with Deep Instrumental Variables Networks
• Multiplex visibility graphs to investigate recurrent neural networks dynamics
• Predicting Citywide Crowd Flows Using Deep Spatio-Temporal Residual Networks
• Applying Deep Learning to Basketball Trajectories
• Visual pathways from the perspective of cost functions and multi-task deep neural networks
• Deepr: A Convolutional Net for Medical Records
• Robustness of Neural Networks against Storage Media Errors
• CHAOS: A Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi
• Historical Document Image Segmentation with LDA-Initialized Deep Neural Networks
• Transport-based analysis, modeling, and learning from signal and data distributions
• Algorithmic Composition of Melodies with Deep Recurrent Neural Networks
• Australia's long-term electricity demand forecasting using deep neural networks
• Spike Event Based Learning in Neural Networks
• Support Spinor Machine
• Quantitative toxicity prediction using topology based multi-task deep neural networks
• Denoising autoencoder with modulated lateral connections learns invariant representations of natural images
• Deep Gaussian Covariance Network
• Feed Forward and Backward Run in Deep Convolution Neural Network
• Dropout with Expectation-linear Regularization
• Gradient-free Policy Architecture Search and Adaptation
• Deep Learning for Automated Quality Assessment of Color Fundus Images in Diabetic Retinopathy Screening
• Using Spatial Pooler of Hierarchical Temporal Memory to classify noisy videos with predefined complexity
• Modelling Competitive Sports: Bradley-Terry-Élő Models for Supervised and On-Line Learning of Paired Competition Outcomes
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Collective Vertex Classification Using Recursive Neural Network
• Transfer of Tool Affordance and Manipulation Cues with 3D Vision Data
• How intelligent are convolutional neural networks?
• Engagement dynamics and sensitivity analysis of YouTube videos
• DeepFeat: A Bottom Up and Top Down Saliency Model Based on Deep Features of Convolutional Neural Nets
• Accelerated Training for Massive Classification via Dynamic Class Selection
• Investigating the Quantum Properties of Jets and the Search for a Supersymmetric Top Quark Partner with the ATLAS Detector
• Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?
• Solutions to problems with deep learning
• Deep Class Aware Denoising
• Transfer entropy-based feedback improves performance in artificial neural networks
• The Future of Data Analysis in the Neurosciences
• Tricks from Deep Learning
• AIXIjs: A Software Demo for General Reinforcement Learning
• Machine Learning and Social Robotics for Detecting Early Signs of Dementia
• Prediction of Prokaryotic and Eukaryotic Promoters Using Convolutional Deep Learning Neural Networks
• Fundamental principles of cortical computation: unsupervised learning with prediction, compression and feedback
• Combining Prediction of Human Decisions with ISMCTS in Imperfect Information Games
• Deep Learning in Robotics: A Review of Recent Research
• A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms
• Towards Building Deep Networks with Bayesian Factor Graphs
• Real-time interactive sequence generation and control with Recurrent Neural Network ensembles
• Deep vs. Diverse Architectures for Classification Problems
• Semi-supervised Learning with Sparse Autoencoders in Phone Classification
• Fine-grained Event Learning of Human-Object Interaction with LSTM-CRF
• Large-scale image analysis using docker sandboxing
• Cognitive Deep Machine Can Train Itself
• Dynamic Key-Value Memory Networks for Knowledge Tracing
• Learning Based on CC1 and CC4 Neural Networks
• Deep Gaussian Mixture Models
• An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning
• Tensor Switching Networks
• Master's Thesis : Deep Learning for Visual Recognition
• A relativistic extension of Hopfield neural networks via the mechanical analogy
• Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach
• Training Feedforward Neural Networks with Standard Logistic Activations is Feasible
• EMR-based medical knowledge representation and inference via Markov random fields and distributed representation learning
• General Backpropagation Algorithm for Training Second-order Neural Networks
• Training Spiking Neural Networks for Cognitive Tasks: A Versatile Framework Compatible to Various Temporal Codes
• Automating Image Analysis by Annotating Landmarks with Deep Neural Networks
• Binary output layer of feedforward neural networks for solving multi-class classification problems
• Lightweight Neural Networks
• Happiness Pursuit: Personality Learning in a Society of Agents
• A generic and fast C++ optimization framework
• What Can Be Predicted from Six Seconds of Driver Glances?
• Learning event representation: As sparse as possible, but not sparser
• Globally Optimal Object Tracking with Fully Convolutional Networks
• Temperature-Based Deep Boltzmann Machines
• Comparison of multi-task convolutional neural network (MT-CNN) and a few other methods for toxicity prediction
• Soft-Deep Boltzmann Machines
• Grounding Hierarchical Reinforcement Learning Models for Knowledge Transfer
• An affective computational model for machine consciousness
• Spiking neurons with short-term synaptic plasticity form superior generative networks
• Intrinsic Geometric Information Transfer Learning on Multiple Graph-Structured Datasets
• Research on Bi-mode Biometrics Based on Deep Learning
• Optimising The Input Window Alignment in CD-DNN Based Phoneme Recognition for Low Latency Processing
• Inferring Mesoscale Models of Neural Computation
• Separation of Water and Fat Magnetic Resonance Imaging Signals Using Deep Learning with Convolutional Neural Networks
• On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience
• A Survey of Brain Inspired Technologies for Engineering
• Learning Robust Representations for Computer Vision
• Fine-Grained Categorization via CNN-Based Automatic Extraction and Integration of Object-Level and Part-Level Features
• RoboCup 2D Soccer Simulation League: Evaluation Challenges