MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task

Abstract

Mathematical formulas are the crystallization of human wisdom in exploring the laws of nature for thousands of years. Describing the complex laws of nature with a concise mathematical formula is a constant pursuit of scientists and a great challenge for artificial intelligence. This field is called symbolic regression (SR). Symbolic regression was originally formulated as a combinatorial optimization problem, and Genetic Programming (GP) and Reinforcement Learning algorithms were used to solve it. However, GP is sensitive to hyperparameters, and these two types of algorithms are inefficient. To solve this problem, researchers treat the mapping from data to expressions as a translation problem. And the corresponding large-scale pre-trained model is introduced. However, the data and expression skeletons do not have very clear word correspondences as the two languages do. Instead, they are more like two modalities (e.g., image and text). Therefore, in this paper, we proposed MMSR. The SR problem is solved as a pure multi-modal problem, and contrastive learning is also introduced in the training process for modal alignment to facilitate later modal feature fusion. It is worth noting that to better promote the modal feature fusion, we adopt the strategy of training contrastive learning loss and other losses at the same time, which only needs one-step training, instead of training contrastive learning loss first and then training other losses. Because our experiments prove training together can make the feature extraction module and feature fusion module wearing-in better. Experimental results show that compared with multiple large-scale pre-training baselines, MMSR achieves the most advanced results on multiple mainstream datasets including SRBench. Our code is open source at https://github.com/1716757342/MMSR

The Information Fusion has accepted this paper

References in corpus (20)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• A Survey of Large Language Models
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Visual Instruction Tuning
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• GLM-130B: An Open Bilingual Pre-trained Model
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• Contemporary Symbolic Regression Methods and their Relative Performance
• SymbolicGPT: A Generative Transformer Model for Symbolic Regression
• PaLI-X: On Scaling up a Multilingual Vision and Language Model
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Neural Symbolic Regression that Scales
• Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering
• Transformer-based Planning for Symbolic Regression
• SymFormer: End-to-end symbolic regression using transformer-based architecture
• Deep Generative Symbolic Regression
• SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training
• MetaSymNet: A Tree-like Symbol Network with Adaptive Architecture and Activation Functions
• Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search

Cited by in corpus (1)

• UniSymNet: A Unified Symbolic Network Guided by Transformer