The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Abstract

Deep learning has demonstrated tremendous success in variety of application domains in the past few years. This new field of machine learning has been growing rapidly and applied in most of the application domains with some new modalities of applications, which helps to open new opportunity. There are different methods have been proposed on different category of learning approaches, which includes supervised, semi-supervised and un-supervised learning. The experimental results show state-of-the-art performance of deep learning over traditional machine learning approaches in the field of Image Processing, Computer Vision, Speech Recognition, Machine Translation, Art, Medical imaging, Medical information processing, Robotics and control, Bio-informatics, Natural Language Processing (NLP), Cyber security, and many more. This report presents a brief survey on development of DL approaches, including Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) including Long Short Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). In addition, we have included recent development of proposed advanced variant DL techniques based on the mentioned DL approaches. Furthermore, DL approaches have explored and evaluated in different application domains are also included in this survey. We have also comprised recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys have published on Deep Learning in Neural Networks [1, 38] and a survey on RL [234]. However, those papers have not discussed the individual advanced techniques for training large scale deep learning models and the recently developed method of generative models [1].

39 pages, 46 figures, 3 tables. arXiv admin note: text overlap with arXiv:1408.3264, arXiv:1411.4046

References in corpus (41)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• A Survey on Deep Learning in Medical Image Analysis
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Playing Atari with Deep Reinforcement Learning
• Striving for Simplicity: The All Convolutional Net
• Generative Adversarial Text to Image Synthesis
• FitNets: Hints for Thin Deep Nets
• Densely Connected Convolutional Networks
• Wide Residual Networks
• An Analysis of Deep Neural Network Models for Practical Applications
• Learning Convolutional Neural Networks for Graphs
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Deep Reinforcement Learning: An Overview
• End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network
• Visualizing and Understanding Convolutional Networks
• Aggregated Residual Transformations for Deep Neural Networks
• A Survey of Neuromorphic Computing and Neural Networks in Hardware
• Xception: Deep Learning with Depthwise Separable Convolutions
• Data Programming: Creating Large Training Sets, Quickly
• Recent Advances in Convolutional Neural Networks
• A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI
• Deep Networks with Stochastic Depth
• One Model To Learn Them All
• All you need is a good init
• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• A Deep Convolutional Neural Network for Background Subtraction
• Segmental Recurrent Neural Networks for End-to-end Speech Recognition
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• Block-Matching Convolutional Neural Network for Image Denoising
• EmptyHeaded: A Relational Engine for Graph Processing
• LCNN: Lookup-based Convolutional Neural Network
• Discriminative Segmental Cascades for Feature-Rich Phone Recognition
• Convolutional Gated Recurrent Neural Network Incorporating Spatial Features for Audio Tagging
• Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
• Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network
• Deep Convolutional Neural Networks for Pairwise Causality
• Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks
• A-Lamp: Adaptive Layout-Aware Multi-Patch Deep Convolutional Neural Network for Photo Aesthetic Assessment

Cited by in corpus (40)

• Deep learning tools for the measurement of animal behavior in neuroscience
• Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation
• Learning New Physics from a Machine
• Multiclass Wound Image Classification using an Ensemble Deep CNN-based Classifier
• The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research
• Review: Deep Learning in Electron Microscopy
• Trends in deep learning for medical hyperspectral image analysis
• A novel database of Children's Spontaneous Facial Expressions (LIRIS-CSE)
• Solving inverse problems using conditional invertible neural networks
• Multilingual Sentiment Analysis: An RNN-Based Framework for Limited Data
• Evaluation of Distance Measures for Feature based Image Registration using AlexNet
• Large image datasets: A pyrrhic win for computer vision?
• An Empirical Study on Writer Identification & Verification from Intra-variable Individual Handwriting
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• Human Vocal Sentiment Analysis
• Platform for Situated Intelligence
• Fooling a Real Car with Adversarial Traffic Signs
• Deep Learning Based Caching for Self-Driving Car in Multi-access Edge Computing
• SWAG: Item Recommendations using Convolutions on Weighted Graphs
• Intra-Variable Handwriting Inspection Reinforced with Idiosyncrasy Analysis
• Multiresolution Convolutional Autoencoders
• SEnSeI: A Deep Learning Module for Creating Sensor Independent Cloud Masks
• Rethinking FUN: Frequency-Domain Utilization Networks
• Artificial Intelligence in Drug Discovery: Applications and Techniques
• Deep Learning Convective Flow Using Conditional Generative Adversarial Networks
• Adversarial Security Attacks and Perturbations on Machine Learning and Deep Learning Methods
• Physical Artificial Intelligence: The Concept Expansion of Next-Generation Artificial Intelligence
• Hardware Implementation of Hyperbolic Tangent Function using Catmull-Rom Spline Interpolation
• Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network
• Double Backpropagation for Training Autoencoders against Adversarial Attack
• Computer vision for liquid samples in hospitals and medical labs using hierarchical image segmentation and relations prediction
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Multiclass Burn Wound Image Classification Using Deep Convolutional Neural Networks
• Networks with pixels embedding: a method to improve noise resistance in images classification
• SplitPlace: Intelligent Placement of Split Neural Nets in Mobile Edge Environments
• R2U3D: Recurrent Residual 3D U-Net for Lung Segmentation
• HODET: Hybrid Object DEtection and Tracking using mmWave Radar and Visual Sensors
• COVID-CLNet: COVID-19 Detection with Compressive Deep Learning Approaches
• Safe Augmentation: Learning Task-Specific Transformations from Data
• Machine Learning for New Physics Searches