Recent Advances in Convolutional Neural Networks

Abstract

In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Among different types of deep neural networks, convolutional neural networks have been most extensively studied. Leveraging on the rapid growth in the amount of the annotated data and the great improvements in the strengths of graphics processor units, the research on convolutional neural networks has been emerged swiftly and achieved state-of-the-art results on various tasks. In this paper, we provide a broad survey of the recent advances in convolutional neural networks. We detailize the improvements of CNN on different aspects, including layer design, activation function, loss function, regularization, optimization and fast computation. Besides, we also introduce various applications of convolutional neural networks in computer vision, speech and natural language processing.

Pattern Recognition, Elsevier

References in corpus (81)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Natural Language Processing (almost) from Scratch
• Caffe: Convolutional Architecture for Fast Feature Embedding
• WaveNet: A Generative Model for Raw Audio
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Generative Adversarial Text to Image Synthesis
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• Understanding Neural Networks Through Deep Visualization
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Training Very Deep Networks
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• cuDNN: Efficient Primitives for Deep Learning
• Compressing Deep Convolutional Networks using Vector Quantization
• Character-Aware Neural Language Models
• Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification
• Exploring the Limits of Language Modeling
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Deep Learning using Linear Support Vector Machines
• The Loss Surfaces of Multilayer Networks
• Predicting Parameters in Deep Learning
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Resnet in Resnet: Generalizing Residual Architectures
• Tensorizing Neural Networks
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Deep Learning for Answer Sentence Selection
• Indoor Semantic Segmentation using depth information
• Scalable, High-Quality Object Detection
• Neural Machine Translation in Linear Time
• Holistically-Nested Edge Detection
• Deep Generative Stochastic Networks Trainable by Backprop
• Fast Training of Convolutional Networks through FFTs
• No More Pesky Learning Rates
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Weakly Supervised Fine-Grained Image Categorization
• Unsupervised Learning of Visual Representations using Videos
• Learning Human Pose Estimation Features with Convolutional Networks
• Feature Learning in Deep Neural Networks - Studies on Speech Recognition Tasks
• Bitwise Neural Networks
• Online Batch Selection for Faster Training of Neural Networks
• Learning to See by Moving
• DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection
• Swapout: Learning an ensemble of deep architectures
• Deep Learning of Representations: Looking Forward
• Combining Local Appearance and Holistic View: Dual-Source Deep Neural Networks for Human Pose Estimation
• Deep Structured Output Learning for Unconstrained Text Recognition
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Denoising Criterion for Variational Auto-Encoding Framework
• GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
• Bilinear CNNs for Fine-grained Visual Recognition
• Fracking Deep Convolutional Image Descriptors
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• DeepHash: Getting Regularization, Depth and Fine-Tuning Right
• Efficient Object Localization Using Convolutional Networks
• Mirror, mirror on the wall, tell me, is the error small?
• DAG-Recurrent Neural Networks For Scene Labeling
• On Accelerated Methods in Optimization
• Very Deep Convolutional Neural Networks for Robust Speech Recognition
• $gen$CNN: A Convolutional Architecture for Word Sequence Prediction
• InterActive: Inter-Layer Activeness Propagation
• Learning Fine-grained Features via a CNN Tree for Large-scale Classification

Cited by in corpus (33)

• A Survey on Deep Learning in Medical Image Analysis
• A systematic study of the class imbalance problem in convolutional neural networks
• Object Detection in 20 Years: A Survey
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• BSNet: Bi-Similarity Network for Few-shot Fine-grained Image Classification
• MASTER: Multi-Aspect Non-local Network for Scene Text Recognition
• SELD-TCN: Sound Event Localization & Detection via Temporal Convolutional Networks
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• Recurrent Attentional Networks for Saliency Detection
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• CASI: A Convolutional Neural Network Approach for Shell Identification
• Deep Learning based Cephalometric Landmark Identification using Landmark-dependent Multi-scale Patches
• A New Dataset and Proposed Convolutional Neural Network Architecture for Classification of American Sign Language Digits
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• An astronomical image content-based recommendation system using combined deep learning models in a fully unsupervised mode
• RapidAI4EO: Mono- and Multi-temporal Deep Learning models for Updating the CORINE Land Cover Product
• Adaptive Convolutional ELM For Concept Drift Handling in Online Stream Data
• Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images
• Distributed Averaging CNN-ELM for Big Data
• Deep Learning for Metagenomic Data: using 2D Embeddings and Convolutional Neural Networks
• Using stigmergy to incorporate the time into artificial neural networks
• Interplay of Traditional Methods and Machine Learning Algorithms for Tagging Boosted Objects
• An Empirical Study of Language CNN for Image Captioning
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• Multilevel Context Representation for Improving Object Recognition
• Master's Thesis : Deep Learning for Visual Recognition
• cuConv: A CUDA Implementation of Convolution for CNN Inference
• A Novel Deep Neural Network Architecture for Mars Visual Navigation
• Microaneurysm Detection in Fundus Images Using a Two-step Convolutional Neural Networks
• Optimising the Input Image to Improve Visual Relationship Detection
• Deep Octonion Networks
• Reproduction of Lateral Inhibition-Inspired Convolutional Neural Network for Visual Attention and Saliency Detection