Max and Coincidence Neurons in Neural Networks
arXiv:2110.01218
Abstract
Network design has been a central topic in machine learning. Large amounts of effort have been devoted towards creating efficient architectures through manual exploration as well as automated neural architecture search. However, todays architectures have yet to consider the diversity of neurons and the existence of neurons with specific processing functions. In this work, we optimize networks containing models of the max and coincidence neurons using neural architecture search, and analyze the structure, operations, and neurons of optimized networks to develop a signal-processing ResNet. The developed network achieves an average of 2% improvement in accuracy and a 25% improvement in network size across a variety of datasets, demonstrating the importance of neuronal functions in creating compact, efficient networks.
References in corpus (10)
- Neural Architecture Search with Reinforcement Learning
- Wide Residual Networks
- Neural Architecture Search: A Survey
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- Efficient Neural Architecture Search via Parameter Sharing
- EMNIST: an extension of MNIST to handwritten letters
- Deep Learning Scaling is Predictable, Empirically
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Efficient Architecture Search by Network Transformation
- Hierarchical Representations for Efficient Architecture Search