Knowledge Matters: Importance of Prior Information for Optimization

Abstract

We explore the effect of introducing prior information into the intermediate level of neural networks for a learning task on which all the state-of-the-art machine learning algorithms tested failed to learn. We motivate our work from the hypothesis that humans learn such intermediate concepts from other individuals via a form of supervision or guidance using a curriculum. The experiments we have conducted provide positive evidence in favor of this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset with 64x64 binary inputs images, each image with three sprites. The final task is to decide whether all the sprites are the same or one of them is different. Sprites are pentomino tetris shapes and they are placed in an image with different locations using scaling and rotation transformations. The first part of the two-tiered MLP is pre-trained with intermediate-level targets being the presence of sprites at each location, while the second part takes the output of the first part as input and predicts the final task's target binary event. The two-tiered MLP architecture, with a few tens of thousand examples, was able to learn the task perfectly, whereas all other algorithms (include unsupervised pre-training, but also traditional algorithms like SVMs, decision trees and boosting) all perform no better than chance. We hypothesize that the optimization difficulty involved when the intermediate pre-training is not performed is due to the {\em composition} of two highly non-linear tasks. Our findings are also consistent with hypotheses on cultural learning inspired by the observations of optimization problems with deep learning, presumably because of effective local minima.

37 Pages, 5 figures, 5 tables JMLR Special Topics on Representation Learning Submission

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Representation Learning: A Review and New Perspectives
• Big Neural Networks Waste Capacity
• Evolving Culture vs Local Minima

Cited by in corpus (20)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• FitNets: Hints for Thin Deep Nets
• Predicting Parameters in Deep Learning
• A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Deep Learning of Representations: Looking Forward
• Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
• Extraction of Salient Sentences from Labelled Documents
• A Deep Multimodal Approach for Cold-start Music Recommendation
• Towards an integration of deep learning and neuroscience
• Emergence of Compositional Language with Deep Generational Transmission
• A Review of Emerging Research Directions in Abstract Visual Reasoning
• Combining Convolution and Recursive Neural Networks for Sentiment Analysis
• Multimodal Transitions for Generative Stochastic Networks
• Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
• Deep Layered Learning in MIR
• A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design
• Learning Abstract Classes using Deep Learning
• DAF-NET: a saliency based weakly supervised method of dual attention fusion for fine-grained image classification
• A New Benchmark and Progress Toward Improved Weakly Supervised Learning