Learning from Higher-Layer Feature Visualizations

Abstract

Driven by the goal to enable sleep apnea monitoring and machine learning-based detection at home with small mobile devices, we investigate whether interpretation-based indirect knowledge transfer can be used to create classifiers with acceptable performance. Interpretation-based indirect knowledge transfer means that a classifier (student) learns from a synthetic dataset based on the knowledge representation from an already trained Deep Network (teacher). We use activation maximization to generate visualizations and create a synthetic dataset to train the student classifier. This approach has the advantage that student classifiers can be trained without access to the original training data. With experiments we investigate the feasibility of interpretation-based indirect knowledge transfer and its limitations. The student achieves an accuracy of 97.8% on MNIST (teacher accuracy: 99.3%) with a similar smaller architecture to that of the teacher. The student classifier achieves an accuracy of 86.1% and 89.5% for a subset of the Apnea-ECG dataset (teacher: 89.5% and 91.1%, respectively).

References in corpus (6)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• Adversarial Transformation Networks: Learning to Generate Adversarial Examples
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks

Cited by in corpus (1)

• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier