Spatially-sparse convolutional neural networks

Abstract

Convolutional neural networks (CNNs) perform well on problems such as handwriting recognition and image classification. However, the performance of the networks is often limited by budget and time constraints, particularly when trying to train deep networks. Motivated by the problem of online handwriting recognition, we developed a CNN for processing spatially-sparse inputs; a character drawn with a one-pixel wide pen on a high resolution grid looks like a sparse matrix. Taking advantage of the sparsity allowed us more efficiently to train and test large, deep CNNs. On the CASIA-OLHWDB1.1 dataset containing 3755 character classes we get a test error of 3.82%. Although pictures are not sparse, they can be thought of as sparse by adding padding. Applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets: 6.28% on CIFAR-10 and 24.30% for CIFAR-100.

13 pages

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (46)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Training Very Deep Networks
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Fractional Max-Pooling
• Grid Long Short-Term Memory
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• All you need is a good init
• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• SMASH: Co-designing Software Compression and Hardware-Accelerated Indexing for Efficient Sparse Matrix Operations
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Recurrent Slice Networks for 3D Segmentation of Point Clouds
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Deep Learning for 3D Point Clouds: A Survey
• More is Less: A More Complicated Network with Less Inference Complexity
• A Bayesian encourages dropout
• OctNet: Learning Deep 3D Representations at High Resolutions
• H-CNN: Spatial Hashing Based CNN for 3D Shape Analysis
• Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
• Deep Linear Discriminant Analysis
• Steganalysis via a Convolutional Neural Network using Large Convolution Filters for Embedding Process with Same Stego Key
• MortonNet: Self-Supervised Learning of Local Features in 3D Point Clouds
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Training Binary Neural Networks using the Bayesian Learning Rule
• NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping
• Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark
• Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures
• CT Perfusion is All We Need: 4D CNN Segmentation of Penumbra and Core in Patients With Suspected Ischemic Stroke
• Geometric features for voxel-based surface recognition
• Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions
• OccuSeg: Occupancy-aware 3D Instance Segmentation
• Unsupervised Data Imputation via Variational Inference of Deep Subspaces
• Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images
• On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Models
• Sketch-Specific Data Augmentation for Freehand Sketch Recognition
• Universum Prescription: Regularization using Unlabeled Data
• Convolutional Tables Ensemble: classification in microseconds
• Realization of spatial sparseness by deep ReLU nets with massive data
• Cirrus: A Long-range Bi-pattern LiDAR Dataset
• Learning with Rethinking: Recurrently Improving Convolutional Neural Networks through Feedback
• SparseDet: Towards End-to-End 3D Object Detection
• Putting 3D Spatially Sparse Networks on a Diet
• Traffic4cast -- Large-scale Traffic Prediction using 3DResNet and Sparse-UNet
• Indoor Scene Recognition in 3D
• C2G-Net: Exploiting Morphological Properties for Image Classification