Neuronal diversity can improve machine learning for physics and beyond

Abstract

Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses. Examples include conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning Hénon-Heiles stellar orbits and the swing of a video recorded pendulum clock. Such \textit{learned diversity} provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artificial systems.

arXiv comment13 pages, 9 figures

References in corpus (12)

• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• A survey on modern trainable activation functions
• Complex systems: physics beyond physics
• Quantum reservoir computing with a single nonlinear oscillator
• Toward an AI Physicist for Unsupervised Learning
• Frequently Asked Questions for: The Atoms of Neural Computation
• Physics enhanced neural networks predict order and chaos
• Learning Combinations of Activation Functions
• Evidence of universality for the May-Wigner stability theorem for random networks with local dynamics
• Adaptable Hamiltonian neural networks
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Weak-winner phase synchronization: A curious case of weak interactions