Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models

Abstract

We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains.

References in corpus (36)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations
• The anti-k_t jet clustering algorithm
• A Brief Introduction to PYTHIA 8.1
• DELPHES 3, A modular framework for fast simulation of a generic collider experiment
• Neural Discrete Representation Learning
• Convolutional Sequence to Sequence Learning
• On the Opportunities and Risks of Foundation Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Flamingo: a Visual Language Model for Few-Shot Learning
• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• Automatic spin-entangled decays of heavy resonances in Monte Carlo simulations
• Classification without labels: Learning from mixed samples in high energy physics
• Extending the Bump Hunt with Machine Learning
• Simulation Assisted Likelihood-free Anomaly Detection
• Dijet resonance search with weak supervision using $\sqrt{s}=13$ TeV $pp$ collisions in the ATLAS detector
• Evaluating generative models in high energy physics
• Point Cloud Transformers applied to Collider Physics
• Symmetries, Safety, and Self-Supervision
• Particle Transformer for Jet Tagging
• FETA: Flow-Enhanced Transportation for Anomaly Detection
• Permutationless Many-Jet Event Reconstruction with Symmetry Preserving Attention Networks
• Learning the language of QCD jets with transformers
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• A method to challenge symmetries in data with self-supervised learning
• Jet Diffusion versus JetGPT -- Modern Networks for the LHC
• Anomalies, Representations, and Self-Supervision
• Towards Galaxy Foundation Models with Hybrid Contrastive Learning
• Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks
• Position Prediction as an Effective Pretraining Strategy
• Pre-training strategy using real particle collision data for event classification in collider physics
• Finetuning Foundation Models for Joint Analysis Optimization
• Differentiable Vertex Fitting for Jet Flavour Tagging

Cited by in corpus (19)

• OmniJet-$\alpha$: The first cross-task foundation model for particle physics
• A Method to Simultaneously Facilitate All Jet Physics Tasks
• Solving Key Challenges in Collider Physics with Foundation Models
• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• SKATR: A Self-Supervised Summary Transformer for SKA
• Weakly supervised anomaly detection for resonant new physics in the dijet final state using proton-proton collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Fine-tuning machine-learned particle-flow reconstruction for new detector geometries in future colliders
• A unified machine learning approach for reconstructing hadronically decaying tau leptons
• Foundations of automatic feature extraction at LHC--point clouds and graphs
• Scalable Multi-Task Learning for Particle Collision Event Reconstruction with Heterogeneous Graph Neural Networks
• Extrapolating Jet Radiation with Autoregressive Transformers
• OmniLearned: A Foundation Model Framework for All Tasks Involving Jet Physics
• Reconstructing hadronically decaying tau leptons with a jet foundation model
• Anomaly preserving contrastive neural embeddings for end-to-end model-independent searches at the LHC
• Tagging fully hadronic exotic decays of the vectorlike $\mathbf{B}$ quark using a graph neural network
• jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
• Strong CWoLa: Binary Classification Without Background Simulation
• Large Language Models -- the Future of Fundamental Physics?