What exactly did the Transformer learn from our physics data?

Abstract

Transformer networks excel in scientific applications. We explore two scenarios in ultra-high-energy cosmic ray simulations to examine what these network architectures learn. First, we investigate the trained positional encodings in air showers which are azimuthally symmetric. Second, we visualize the attention values assigned to cosmic particles originating from a galaxy catalog. In both cases, the Transformers learn plausible, physically meaningful features.

9 pages, 7 figures

References in corpus (13)

• HEALPix -- a Framework for High Resolution Discretization, and Fast Analysis of Data Distributed on the Sphere
• The Pierre Auger Cosmic Ray Observatory
• The Galactic Magnetic Field
• Deriving global structure of the Galactic Magnetic Field from Faraday Rotation Measures of extragalactic sources
• CRPropa 3.2 -- an advanced framework for high-energy particle propagation in extragalactic and galactic spaces
• The Coherent Magnetic Field of the Milky Way
• Inference of cosmic-ray source properties by conditional invertible neural networks
• Constraining models for the origin of ultra-high-energy cosmic rays with a novel combined analysis of arrival directions, spectrum, and composition data measured at the Pierre Auger Observatory
• Learning the language of QCD jets with transformers
• Deep-Learning based Reconstruction of the Shower Maximum $X_{\max }$ using the Water-Cherenkov Detectors of the Pierre Auger Observatory
• Extraction of the Muon Signals Recorded with the Surface Detector of the Pierre Auger Observatory Using Recurrent Neural Networks
• Measurement of the Depth of Maximum of Air-Shower Profiles with energies between $10^{18.5}$ and $10^{20}$ eV using the Surface Detector of the Pierre Auger Observatory and Deep Learning
• Jet Tagging with More-Interaction Particle Transformer

Cited by in corpus (1)

• The Physics Behind ML-based Quark-Gluon Taggers