Parallel Hyperparameter Optimization Of Spiking Neural Network

Abstract

Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very sensitive to the hyperparameters, making their optimization challenging. To tackle hyperparameter optimization of SNNs, we initially extended the signal loss issue of SNNs to what we call silent networks. These networks fail to emit enough spikes at their outputs due to mistuned hyperparameters or architecture. Generally, search spaces are heavily restrained, sometimes even discretized, to prevent the sampling of such networks. By defining an early stopping criterion detecting silent networks and by designing specific constraints, we were able to instantiate larger and more flexible search spaces. We applied a constrained Bayesian optimization technique, which was asynchronously parallelized, as the evaluation time of a SNN is highly stochastic. Large-scale experiments were carried-out on a multi-GPU Petascale architecture. By leveraging silent networks, results show an acceleration of the search, while maintaining good performances of both the optimization algorithm and the best solution obtained. We were able to apply our methodology to two popular training algorithms, known as spike timing dependent plasticity and surrogate gradient. Early detection allowed us to prevent worthless and costly computation, directing the search toward promising hyperparameter combinations. Our methodology could be applied to multi-objective problems, where the spiking activity is often minimized to reduce the energy consumption. In this scenario, it becomes essential to find the delicate frontier between low-spiking and silent networks. Finally, our approach may have implications for neural architecture search, particularly in defining suitable spiking architectures.

References in corpus (16)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• SLAYER: Spike Layer Error Reassignment in Time
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• BindsNET: A machine learning-oriented spiking neural networks library in Python
• Surrogate Gradient Learning in Spiking Neural Networks
• A new Taxonomy of Continuous Global Optimization Algorithms
• Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space
• Parameter Optimization and Learning in a Spiking Neural Network for UAV Obstacle Avoidance targeting Neuromorphic Processors
• Differential Evolution and Bayesian Optimisation for Hyper-Parameter Selection in Mixed-Signal Neuromorphic Circuits Applied to UAV Obstacle Avoidance
• Unsupervised Visual Feature Learning with Spike-timing-dependent Plasticity: How Far are we from Traditional Feature Learning Approaches?
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks
• AutoSNN: Towards Energy-Efficient Spiking Neural Networks
• Scalable Constrained Bayesian Optimization
• NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems
• A Comprehensive Review of Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practices
• Adaptive control for hindlimb locomotion in a simulated mouse through temporal cerebellar learning