OmniJet-$\alpha$: The first cross-task foundation model for particle physics

Abstract

Foundation models are multi-dataset and multi-task machine learning methods that once pre-trained can be fine-tuned for a large variety of downstream applications. The successful development of such general-purpose models for physics data would be a major breakthrough as they could improve the achievable physics performance while at the same time drastically reduce the required amount of training time and data. We report significant progress on this challenge on several fronts. First, a comprehensive set of evaluation methods is introduced to judge the quality of an encoding from physics data into a representation suitable for the autoregressive generation of particle jets with transformer architectures (the common backbone of foundation models). These measures motivate the choice of a higher-fidelity tokenization compared to previous works. Finally, we demonstrate transfer learning between an unsupervised problem (jet generation) and a classic supervised task (jet tagging) with our new OmniJet-$\alpha$ model. This is the first successful transfer between two different and actively studied classes of tasks and constitutes a major step in the building of foundation models for particle physics.

References in corpus (4)

• The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations
• An Introduction to PYTHIA 8.2
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• Classifier Surrogates: Sharing AI-based Searches with the World

Cited by in corpus (19)

• A Method to Simultaneously Facilitate All Jet Physics Tasks
• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• Convolutional L2LFlows: Generating Accurate Showers in Highly Granular Calorimeters Using Convolutional Normalizing Flows
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics
• SKATR: A Self-Supervised Summary Transformer for SKA
• Advancing Set-Conditional Set Generation: Diffusion Models for Fast Simulation of Reconstructed Particles
• Fine-tuning machine-learned particle-flow reconstruction for new detector geometries in future colliders
• Foundations of automatic feature extraction at LHC--point clouds and graphs
• BitHEP -- The Limits of Low-Precision ML in HEP
• Scalable Multi-Task Learning for Particle Collision Event Reconstruction with Heterogeneous Graph Neural Networks
• Extrapolating Jet Radiation with Autoregressive Transformers
• Anomaly preserving contrastive neural embeddings for end-to-end model-independent searches at the LHC
• OmniLearned: A Foundation Model Framework for All Tasks Involving Jet Physics
• Reconstructing hadronically decaying tau leptons with a jet foundation model
• Machine Learning Electroweakino Production
• jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
• Large Language Models -- the Future of Fundamental Physics?
• Stable and Interpretable Jet Physics with IRC-Safe Equivariant Feature Extraction