How far can we go without convolution: Improving fully-connected networks

Abstract

We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is just 10% short of a decent convolutional network.

10 pages, 11 figures, submitted for ICLR 2016

References in corpus (1)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Cited by in corpus (11)

• Biologically plausible deep learning -- but how far can we go with shallow networks?
• Artificial Intelligence and Deep Learning Algorithms for Epigenetic Sequence Analysis: A Review for Epigeneticists and AI Experts
• Variational Information Distillation for Knowledge Transfer
• Computational Separation Between Convolutional and Fully-Connected Networks
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Exploring the Back Alleys: Analysing The Robustness of Alternative Neural Network Architectures against Adversarial Attacks
• Accelerating Fully Connected Neural Network on Optical Network-on-Chip (ONoC)
• Cascaded Classifier for Pareto-Optimal Accuracy-Cost Trade-Off Using off-the-Shelf ANNs
• Intelligent Matrix Exponentiation
• Group-Connected Multilayer Perceptron Networks
• Graphs for deep learning representations