Stacking-Based Deep Neural Network: Deep Analytic Network for Pattern Classification

Abstract

Stacking-based deep neural network (S-DNN) is aggregated with pluralities of basic learning modules, one after another, to synthesize a deep neural network (DNN) alternative for pattern classification. Contrary to the DNNs trained end to end by backpropagation (BP), each S-DNN layer, i.e., a self-learnable module, is to be trained decisively and independently without BP intervention. In this paper, a ridge regression-based S-DNN, dubbed deep analytic network (DAN), along with its kernelization (K-DAN), are devised for multilayer feature re-learning from the pre-extracted baseline features and the structured features. Our theoretical formulation demonstrates that DAN/K-DAN re-learn by perturbing the intra/inter-class variations, apart from diminishing the prediction errors. We scrutinize the DAN/K-DAN performance for pattern classification on datasets of varying domains - faces, handwritten digits, generic objects, to name a few. Unlike the typical BP-optimized DNNs to be trained from gigantic datasets by GPU, we disclose that DAN/K-DAN are trainable using only CPU even for small-scale training sets. Our experimental results disclose that DAN/K-DAN outperform the present S-DNNs and also the BP-trained DNNs, including multiplayer perceptron, deep belief network, etc., without data augmentation applied.

arXiv comment14 pages, 7 figures, 11 tables

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Deep Residual Learning for Image Recognition
• How transferable are features in deep neural networks?
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Striving for Simplicity: The All Convolutional Net
• PCANet: A Simple Deep Learning Baseline for Image Classification?
• DeepID3: Face Recognition with Very Deep Neural Networks
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Fractional Max-Pooling
• Multi-Fold Gabor, PCA and ICA Filter Convolution Descriptor for Face Recognition
• Forward Thinking: Building and Training Neural Networks One Layer at a Time

Cited by in corpus (1)

• Ensemble deep learning: A review