On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case

Abstract

Beginning from a basic neural-network architecture, we test the potential benefits offered by a range of advanced techniques for machine learning, in particular deep learning, in the context of a typical classification problem encountered in the domain of high-energy physics, using a well-studied dataset: the 2014 Higgs ML Kaggle dataset. The advantages are evaluated in terms of both performance metrics and the time required to train and apply the resulting models. Techniques examined include domain-specific data-augmentation, learning rate and momentum scheduling, (advanced) ensembling in both model-space and weight-space, and alternative architectures and connection methods. Following the investigation, we arrive at a model which achieves equal performance to the winning solution of the original Kaggle challenge, whilst being significantly quicker to train and apply, and being suitable for use with both GPU and CPU hardware setups. These reductions in timing and hardware requirements potentially allow the use of more powerful algorithms in HEP analyses, where models must be retrained frequently, sometimes at short notice, by small groups of researchers with limited hardware resources. Additionally, a new wrapper library for PyTorch called LUMIN is presented, which incorporates all of the techniques studied.

arXiv commentPreprint V4: Fixing typographical error and correcting two plots. Mach. Learn.: Sci. Technol (2020)

References in corpus (39)

• Scikit-learn: Machine Learning in Python
• XGBoost: A Scalable Tree Boosting System
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• The NumPy array: a structure for efficient numerical computation
• Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
• Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC
• Asymptotic formulae for likelihood-based tests of new physics
• DELPHES 3, A modular framework for fast simulation of a generic collider experiment
• Squeeze-and-Excitation Networks
• Densely Connected Convolutional Networks
• Deeply-Supervised Nets
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• General-purpose event generators for LHC physics
• Deep Learning and its Application to LHC Physics
• CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks
• Entity Embeddings of Categorical Variables
• Wide & Deep Learning for Recommender Systems
• The Machine Learning Landscape of Top Taggers
• Parameterized Machine Learning for High-Energy Physics
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• A New Mass Reconstruction Technique for Resonances Decaying to di-tau
• Regularization for Deep Learning: A Taxonomy
• Essentially No Barriers in Neural Network Energy Landscape
• Mining gold from implicit models to improve likelihood-free inference
• Adversarially-trained autoencoders for robust unsupervised new physics searches
• Measurement of the t-channel single top quark production cross section in pp collisions at sqrt(s) = 7 TeV
• JEDI-net: a jet identification algorithm based on interaction networks
• QCD-Aware Recursive Neural Networks for Jet Physics
• Design and performance of the LHCb trigger and full real-time reconstruction in Run 2 of the LHC
• Snapshot Ensembles: Train 1, get M for free
• Rethinking ImageNet Pre-training
• Fast and accurate simulation of particle detectors using generative adversarial networks
• Machine Learning Uncertainties with Adversarial Neural Networks
• Search for Single Top Quark Production at DZero Using Neural Networks
• INFERNO: Inference-Aware Neural Optimisation
• Machine Learning in High Energy Physics Community White Paper
• SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data
• End-to-End Physics Event Classification with CMS Open Data: Applying Image-Based Deep Learning to Detector Data for the Direct Classification of Collision Events at the LHC

Cited by in corpus (7)

• Boosted decision trees in the era of new physics: a smuon analysis case study
• Search for nonresonant Higgs boson pair production in final state with two bottom quarks and two tau leptons in proton-proton collisions at $\sqrt{s}$ = 13 TeV
• Self-Supervised Learning on MeerKAT Wide-Field Continuum Images
• Advances in Multi-Variate Analysis Methods for New Physics Searches at the Large Hadron Collider
• Improving smuon searches with Neural Networks
• How to use Machine Learning to improve the discrimination between signal and background at particle colliders
• Probing new physics in the Boosted $HH\to b\bar{b}\gamma \gamma$ channel at the LHC