Evaluating generative models in high energy physics

Abstract

There has been a recent explosion in research into machine-learning-based generative modeling to tackle computational challenges for simulations in high energy physics (HEP). In order to use such alternative simulators in practice, we need well-defined metrics to compare different generative models and evaluate their discrepancy from the true distributions. We present the first systematic review and investigation into evaluation metrics and their sensitivity to failure modes of generative models, using the framework of two-sample goodness-of-fit testing, and their relevance and viability for HEP. Inspired by previous work in both physics and computer vision, we propose two new metrics, the Fréchet and kernel physics distances (FPD and KPD, respectively), and perform a variety of experiments measuring their performance on simple Gaussian-distributed, and simulated high energy jet datasets. We find FPD, in particular, to be the most sensitive metric to all alternative jet distributions tested and recommend its adoption, along with the KPD and Wasserstein distances between individual feature distributions, for evaluating generative models in HEP. We finally demonstrate the efficacy of these proposed metrics in evaluating and comparing a novel attention-based generative adversarial particle transformer to the state-of-the-art message-passing generative adversarial network jet simulation model. The code for our proposed metrics is provided in the open source JetNet Python library.

11 pages, 5 figures, 3 tables, and a 5 page appendix

Cited by in corpus (26)

• CaloClouds: Fast Geometry-Independent Highly-Granular Calorimeter Simulation
• EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets
• PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics
• L2LFlows: Generating High-Fidelity 3D Calorimeter Images
• How to Understand Limitations of Generative Networks
• Deep generative models for fast photon shower simulation in ATLAS
• Learning the language of QCD jets with transformers
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• An implementation of neural simulation-based inference for parameter estimation in ATLAS
• Convolutional L2LFlows: Generating Accurate Showers in Highly Granular Calorimeters Using Convolutional Normalizing Flows
• Generalizing to new geometries with Geometry-Aware Autoregressive Models (GAAMs) for fast calorimeter simulation
• Goodness of fit by Neyman-Pearson testing
• Comparison of Affine and Rational Quadratic Spline Coupling and Autoregressive Flows through Robust Statistical Tests
• End-to-end simulation of particle physics events with Flow Matching and generator Oversampling
• CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation
• Differentiable Earth Mover's Distance for Data Compression at the High-Luminosity LHC
• Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics
• Galaxy Imaging with Generative Models: Insights from a Two-Models Framework
• Flow Matching Beyond Kinematics: Generating Jets with Particle-ID and Trajectory Displacement Information
• Refining fast simulation using machine learning
• LHC Hadronic Jet Generation Using Convolutional Variational Autoencoders with Normalizing Flows
• AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning
• BitHEP -- The Limits of Low-Precision ML in HEP
• Event generation and consistency tests with sliced Wasserstein distance in high-energy physics
• Choose Your Diffusion: Efficient and flexible ways to accelerate the diffusion model in fast high energy physics simulation