Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models

Abstract

Self-Supervised Learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose RS3L ("Re-simulation-based self-supervised representation learning"), a novel simulation-based SSL strategy that employs a method of re-simulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and re-running simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pre-training enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.

14 pages, 8 figures

References in corpus (25)

• The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations
• The anti-k_t jet clustering algorithm
• An Introduction to PYTHIA 8.2
• DELPHES 3, A modular framework for fast simulation of a generic collider experiment
• Parton distributions for the LHC Run II
• Extraction and validation of a new set of CMS PYTHIA8 tunes from underlying-event measurements
• Observation of Higgs boson decay to bottom quarks
• Observation of $H\to b\bar{b}$ decays and $VH$ production with the ATLAS detector
• New Angles on Energy Correlation Functions
• Graph Neural Networks in Particle Physics
• Measurements of $WH$ and $ZH$ production in the $H\to b\bar{b}$ decay channel in $pp$ collisions at 13 TeV with the ATLAS detector
• Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at sqrt(s) = 13 TeV
• Event Generators for High-Energy Physics Experiments
• AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• Measurement of the associated production of a Higgs boson decaying into $b$-quarks with a vector boson at high transverse momentum in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for dark matter produced in association with a Higgs boson decaying to a pair of bottom quarks in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Symmetries, Safety, and Self-Supervision
• Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Inclusive search for highly boosted Higgs bosons decaying to bottom quark-antiquark pairs in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• OmniJet-$\alpha$: The first cross-task foundation model for particle physics
• Search for dark matter particles produced in association with a Higgs boson in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• Pile-Up Mitigation using Attention
• Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
• Measurement of simplified template cross sections of the Higgs boson produced in association with W or Z bosons in the H $\to$ $\mathrm{b}\bar{\mathrm{b}}$ decay channel in proton-proton collisions at $\sqrt{s}$ = 13 TeV

Cited by in corpus (12)

• A Method to Simultaneously Facilitate All Jet Physics Tasks
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• Fine-tuning machine-learned particle-flow reconstruction for new detector geometries in future colliders
• MACK: Mismodeling Addressed with Contrastive Knowledge
• BitHEP -- The Limits of Low-Precision ML in HEP
• Contrastive Learning for Robust Representations of Neutrino Data
• Anomaly preserving contrastive neural embeddings for end-to-end model-independent searches at the LHC
• Reconstructing hadronically decaying tau leptons with a jet foundation model
• OmniLearned: A Foundation Model Framework for All Tasks Involving Jet Physics
• Large Language Models -- the Future of Fundamental Physics?
• jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
• Enhancing generalization in high energy physics using white-box adversarial attacks