SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression
arXiv:2401.09949 · doi:10.1088/2632-2153/adaad8
Abstract
Compact symbolic expressions have been shown to be more efficient than neural network models in terms of resource consumption and inference speed when implemented on custom hardware such as FPGAs, while maintaining comparable accuracy~\cite{tsoi2023symbolic}. These capabilities are highly valuable in environments with stringent computational resource constraints, such as high-energy physics experiments at the CERN Large Hadron Collider. However, finding compact expressions for high-dimensional datasets remains challenging due to the inherent limitations of genetic programming, the search algorithm of most symbolic regression methods. Contrary to genetic programming, the neural network approach to symbolic regression offers scalability to high-dimensional inputs and leverages gradient methods for faster equation searching. Common ways of constraining expression complexity often involve multistage pruning with fine-tuning, which can result in significant performance loss. In this work, we propose , a neural network approach to symbolic regression specifically designed as a model compression technique, aimed at enabling low-latency inference for high-dimensional inputs on custom hardware such as FPGAs. This framework allows dynamic pruning of model weights, input features, and mathematical operators in a single training process, where both training loss and expression complexity are optimized simultaneously. We introduce a sparsity regularization term for each pruning type, which can adaptively adjust its strength, leading to convergence at a target sparsity ratio. Unlike most existing symbolic regression methods that struggle with datasets containing more than inputs, we demonstrate the effectiveness of our model on the LHC jet tagging task (16 inputs), MNIST (784 inputs), and SVHN (3072 inputs).
21 pages, 9 figures. to be published in MLST
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Deep Learning with Limited Numerical Precision
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Fast inference of deep neural networks in FPGAs for particle physics
- Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
- PySINDy: A comprehensive Python package for robust sparse system identification
- Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl
- Contemporary Symbolic Regression Methods and their Relative Performance
- Fast convolutional neural networks on FPGAs with hls4ml
- JEDI-net: a jet identification algorithm based on interaction networks
- Improving Model-based Genetic Programming for Symbolic Regression of Small Expressions
- A computational framework for physics-informed symbolic regression with straightforward integration of domain knowledge
- SymbolicGPT: A Generative Transformer Model for Symbolic Regression
- Finding universal relations in subhalo properties with artificial intelligence
- Symbolic Regression is NP-hard
- Neural Symbolic Regression that Scales
- Augmenting astrophysical scaling relations with machine learning: application to reducing the Sunyaev-Zeldovich flux-mass scatter
- The SZ flux-mass (-) relation at low halo masses: improvements with symbolic regression and strong constraints on baryonic feedback
- The importance of calorimetry for highly-boosted jet substructure
- Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression
- Modeling assembly bias with machine learning and symbolic regression
- Symbolic Regression on FPGAs for Fast Machine Learning Inference
- SymFormer: End-to-end symbolic regression using transformer-based architecture
- Informed Equation Learning
- Bayesian Learning to Discover Mathematical Operations in Governing Equations of Dynamic Systems
- Differentiable Genetic Programming for High-dimensional Symbolic Regression