Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Abstract

Good old on-line back-propagation for plain multi-layer perceptrons yields a very low 0.35% error rate on the famous MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images, and graphics cards to greatly speed up learning.

14 pages, 2 figures, 4 listings

Cited by in corpus (120)

• Deep Learning in Neural Networks: An Overview
• Improving neural networks by preventing co-adaptation of feature detectors
• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• A Review of Vibration-Based Damage Detection in Civil Structures: From Traditional Methods to Machine Learning and Deep Learning Applications
• Memory and information processing in neuromorphic systems
• Three dimensional Deep Learning approach for remote sensing image classification
• Machine Learning of Molecular Electronic Properties in Chemical Compound Space
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Deep Representations for Iris, Face, and Fingerprint Spoofing Detection
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Deep Image: Scaling up Image Recognition
• Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
• Compute Trends Across Three Eras of Machine Learning
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Bayesian Neural Networks: An Introduction and Survey
• A Deep Learning Interpretable Classifier for Diabetic Retinopathy Disease Grading
• Building high-level features using large scale unsupervised learning
• Quantum machine learning for image classification
• Neural Network Applications in Earthquake Prediction (1994-2019): Meta-Analytic Insight on their Limitations
• Knowledge Matters: Importance of Prior Information for Optimization
• Further advantages of data augmentation on convolutional neural networks
• Handwritten Isolated Bangla Compound Character Recognition: a new benchmark using a novel deep learning approach
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Super-Resolution via Deep Learning
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Review: Deep Learning in Electron Microscopy
• APAC: Augmented PAttern Classification with Neural Networks
• Representation Learning: A Review and New Perspectives
• SKYNET: an efficient and robust neural network training tool for machine learning in astronomy
• A Kernel Theory of Modern Data Augmentation
• Random feedback weights support learning in deep neural networks
• Artificial Intelligence and its Role in Near Future
• Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm
• Applications and Techniques for Fast Machine Learning in Science
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Multi-column Deep Neural Networks for Image Classification
• TextCaps : Handwritten Character Recognition with Very Small Datasets
• CNN Architecture Comparison for Radio Galaxy Classification
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Photonic Delay Systems as Machine Learning Implementations
• Sparse Activity and Sparse Connectivity in Supervised Learning
• Learning Through Time in the Thalamocortical Loops
• Deep Predictive Learning: A Comprehensive Model of Three Visual Streams
• A Shape Transformation-based Dataset Augmentation Framework for Pedestrian Detection
• Throughput Optimizations for FPGA-based Deep Neural Network Inference
• A Survey of Deep Learning Techniques for Mobile Robot Applications
• Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs
• Towards out of distribution generalization for problems in mechanics
• PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference
• Deep Learning in Multi-Layer Architectures of Dense Nuclei
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method
• Adapting Resilient Propagation for Deep Learning
• CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU
• Over-Sampling in a Deep Neural Network
• Memorized Sparse Backpropagation
• Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network
• Abstract Learning via Demodulation in a Deep Neural Network
• Real-time PCG Anomaly Detection by Adaptive 1D Convolutional Neural Networks
• Distributed Semi-supervised Fuzzy Regression with Interpolation Consistency Regularization
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Open Set Recognition with Conditional Probabilistic Generative Models
• An Energy-efficient Time-domain Analog VLSI Neural Network Processor Based on a Pulse-width Modulation Approach
• Catalog-free modeling of galaxy types in deep images: Massive dimensional reduction with neural networks
• Classification-Reconstruction Learning for Open-Set Recognition
• Cascaded Face Alignment via Intimacy Definition Feature
• ShareBoost: Efficient Multiclass Learning with Feature Sharing
• Robustness of Visual Explanations to Common Data Augmentation
• Deploy Large-Scale Deep Neural Networks in Resource Constrained IoT Devices with Local Quantization Region
• SampleAhead: Online Classifier-Sampler Communication for Learning from Synthesized Data
• My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013
• Topic Space Trajectories: A case study on machine learning literature
• Streaming Object Detection for 3-D Point Clouds
• Accelerating Training of Deep Neural Networks via Sparse Edge Processing
• Learned versus Hand-Designed Feature Representations for 3d Agglomeration
• Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
• SeismoGen: Seismic Waveform Synthesis Using Generative Adversarial Networks
• Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data
• Straight to Shapes++: Real-time Instance Segmentation Made More Accurate
• Accelerating Fully Connected Neural Network on Optical Network-on-Chip (ONoC)
• SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders
• Efficient Training of Convolutional Neural Nets on Large Distributed Systems
• Deep-ER: Deep Learning ECCENTRIC Reconstruction for fast high-resolution neurometabolic imaging
• DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future
• Single Person Pose Estimation: A Survey
• Dropout Rademacher Complexity of Deep Neural Networks
• Transfer Incremental Learning using Data Augmentation
• On the Long-Term Memory of Deep Recurrent Networks
• Introducing a new high-resolution handwritten digits data set with writer characteristics
• Realization of spatial sparseness by deep ReLU nets with massive data
• Deep Learning: Our Miraculous Year 1990-1991
• Bengali Handwritten Grapheme Classification: Deep Learning Approach
• Injectivity, stability, and positive definiteness of max filtering
• Noise Benefits in Expectation-Maximization Algorithms
• Embedding physical symmetries into machine-learned reduced plasma physics models via data augmentation
• Deep Transform: Error Correction via Probabilistic Re-Synthesis
• Learning in Deep Neural Networks Using a Biologically Inspired Optimizer
• On the Effects of Knowledge-Augmented Data in Word Embeddings
• Approximation smooth and sparse functions by deep neural networks without saturation
• Deep Learning At Scale and At Ease
• CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks
• Learning Discriminative Features using Encoder-Decoder type Deep Neural Nets
• Digit Recognition Using Convolution Neural Network
• A parallel Fortran framework for neural networks and deep learning
• Real-time interactive magnetic resonance (MR) temperature imaging in both aqueous and adipose tissues using cascaded deep neural networks for MR-guided focused ultrasound surgery (MRgFUS)
• Towards the Improvement of Automated Scientific Document Categorization by Deep Learning
• Convolutional Neural Network with Pruning Method for Handwritten Digit Recognition
• Bridging the Band Gap: What Device Physicists Need to Know About Machine Learning
• Basic Thresholding Classification
• ${\bar{G}}_{mst}$:An Unbiased Stratified Statistic and a Fast Gradient Optimization Algorithm Based on It
• Label Denoising with Large Ensembles of Heterogeneous Neural Networks
• Deep Transform: Time-Domain Audio Error Correction via Probabilistic Re-Synthesis
• Towards co-evolution of fitness predictors and Deep Neural Networks
• Noisy Feature Mixup
• Deep Double Sparsity Encoder: Learning to Sparsify Not Only Features But Also Parameters
• DNNs as Layers of Cooperating Classifiers
• Partitioning Large Scale Deep Belief Networks Using Dropout
• Semi-supervised learning method based on predefined evenly-distributed class centroids