OmniJet-$\alpha$: The first cross-task foundation model for particle physics

Abstract

Foundation models are multi-dataset and multi-task machine learning methods that once pre-trained can be fine-tuned for a large variety of downstream applications. The successful development of such general-purpose models for physics data would be a major breakthrough as they could improve the achievable physics performance while at the same time drastically reduce the required amount of training time and data. We report significant progress on this challenge on several fronts. First, a comprehensive set of evaluation methods is introduced to judge the quality of an encoding from physics data into a representation suitable for the autoregressive generation of particle jets with transformer architectures (the common backbone of foundation models). These measures motivate the choice of a higher-fidelity tokenization compared to previous works. Finally, we demonstrate transfer learning between an unsupervised problem (jet generation) and a classic supervised task (jet tagging) with our new OmniJet-$\alpha$ model. This is the first successful transfer between two different and actively studied classes of tasks and constitutes a major step in the building of foundation models for particle physics.

References in corpus (27)

• The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations
• The anti-k_t jet clustering algorithm
• An Introduction to PYTHIA 8.2
• DELPHES 3, A modular framework for fast simulation of a generic collider experiment
• Identifying Boosted Objects with N-subjettiness
• Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis
• Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multi-Layer Calorimeters
• Deep-learning Top Taggers or The End of QCD?
• Energy Flow Networks: Deep Sets for Particle Jets
• The Machine Learning Landscape of Top Taggers
• A Roadmap for HEP Software and Computing R&D for the 2020s
• How to GAN LHC Events
• Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
• Pulling Out All the Tops with Computer Vision and Deep Learning
• Machine Learning and LHC Event Generation
• CaloFlow: Fast and Accurate Generation of Calorimeter Showers with Normalizing Flows
• CaloClouds: Fast Geometry-Independent Highly-Granular Calorimeter Simulation
• EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets
• Symmetries, Safety, and Self-Supervision
• Self-supervised Anomaly Detection for New Physics
• Learning the language of QCD jets with transformers
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• Shared Data and Algorithms for Deep Learning in Fundamental Physics
• Neural Embedding: Learning the Embedding of the Manifold of Physics Data
• Generalizing to new geometries with Geometry-Aware Autoregressive Models (GAAMs) for fast calorimeter simulation
• Meta-learning and data augmentation for mass-generalised jet taggers
• Classifier Surrogates: Sharing AI-based Searches with the World

Cited by in corpus (19)

• A Method to Simultaneously Facilitate All Jet Physics Tasks
• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• Convolutional L2LFlows: Generating Accurate Showers in Highly Granular Calorimeters Using Convolutional Normalizing Flows
• Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
• Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics
• SKATR: A Self-Supervised Summary Transformer for SKA
• Advancing Set-Conditional Set Generation: Diffusion Models for Fast Simulation of Reconstructed Particles
• Fine-tuning machine-learned particle-flow reconstruction for new detector geometries in future colliders
• Foundations of automatic feature extraction at LHC--point clouds and graphs
• BitHEP -- The Limits of Low-Precision ML in HEP
• Scalable Multi-Task Learning for Particle Collision Event Reconstruction with Heterogeneous Graph Neural Networks
• Extrapolating Jet Radiation with Autoregressive Transformers
• Anomaly preserving contrastive neural embeddings for end-to-end model-independent searches at the LHC
• OmniLearned: A Foundation Model Framework for All Tasks Involving Jet Physics
• Reconstructing hadronically decaying tau leptons with a jet foundation model
• Machine Learning Electroweakino Production
• jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
• Large Language Models -- the Future of Fundamental Physics?
• Stable and Interpretable Jet Physics with IRC-Safe Equivariant Feature Extraction