Abstract Learning via Demodulation in a Deep Neural Network

Abstract

Inspired by the brain, deep neural networks (DNN) are thought to learn abstract representations through their hierarchical architecture. However, at present, how this happens is not well understood. Here, we demonstrate that DNN learn abstract representations by a process of demodulation. We introduce a biased sigmoid activation function and use it to show that DNN learn and perform better when optimized for demodulation. Our findings constitute the first unambiguous evidence that DNN perform abstract learning in practical use. Our findings may also explain abstract learning in the human brain.

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Over-Sampling in a Deep Neural Network

Cited by in corpus (5)

• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
• Dither is Better than Dropout for Regularising Deep Neural Networks
• "Oddball SGD": Novelty Driven Stochastic Gradient Descent for Training Deep Neural Networks
• Deep Transform: Time-Domain Audio Error Correction via Probabilistic Re-Synthesis