jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation

Abstract

Self-supervised learning, in the context of foundation model training, is a powerful pre-training method for learning feature representations without labels, which often capture generic underlying semantics from the data and can later be fine-tuned for downstream tasks. In this work, we introduce jBOT, a pre-training method based on self-distillation for jet data from the CERN Large Hadron Collider, which combines local particle-level distillation with global jet-level distillation to learn jet representations that support downstream tasks such as anomaly detection and classification. We observe that pre-training on unlabeled jets leads to emergent semantic class clustering in the representation space. The clustering in the frozen embedding, when pre-trained on background jets only, enables anomaly detection via simple distance-based metrics, and the learned embedding can be fine-tuned for classification with improved performance compared to supervised models trained from scratch.

Published in SciPost Phys

References in corpus (28)

• Distilling the Knowledge in a Neural Network
• The anti-k_t jet clustering algorithm
• Gaussian Error Linear Units (GELUs)
• ParticleNet: Jet Tagging via Particle Clouds
• Energy Flow Networks: Deep Sets for Particle Jets
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Graph Neural Networks in Particle Physics
• JEDI-net: a jet identification algorithm based on interaction networks
• An Efficient Lorentz Equivariant Graph Neural Network for Jet Tagging
• Symmetries, Safety, and Self-Supervision
• Particle Transformer for Jet Tagging
• DINOv3
• OmniJet-$\alpha$: The first cross-task foundation model for particle physics
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• Lorentz group equivariant autoencoders
• A Method to Simultaneously Facilitate All Jet Physics Tasks
• Solving Key Challenges in Collider Physics with Foundation Models
• A Lorentz-Equivariant Transformer for All of the LHC
• Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
• CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet
• The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster
• Does Lorentz-symmetric design boost network performance in jet physics?
• Semi-visible jets, energy-based models, and self-supervision
• Particle Cloud Generation with Message Passing Generative Adversarial Networks
• Bumblebee: Foundation Model for Particle Physics Discovery
• MACK: Mismodeling Addressed with Contrastive Knowledge
• RINO: Renormalization Group Invariance with No Labels
• Foundation models for high-energy physics