Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data

Abstract

This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures. We develop supervised convolutional architectures for discriminating signals in high-energy physics data as well as semi-supervised architectures for localizing and classifying extreme weather in climate data. Our Intelcaffe-based implementation obtains $\sim$2TFLOP/s on a single Cori Phase-II Xeon-Phi node. We use a hybrid strategy employing synchronous node-groups, while using asynchronous communication across groups. We use this strategy to scale training of a single model to $\sim$9600 Xeon-Phi nodes; obtaining peak performance of 11.73-15.07 PFLOP/s and sustained performance of 11.41-13.27 PFLOP/s. At scale, our HEP architecture produces state-of-the-art classification accuracy on a dataset with 10M images, exceeding that achieved by selections on high-level physics-motivated features. Our semi-supervised architecture successfully extracts weather patterns in a 15TB climate dataset. Our results demonstrate that Deep Learning can be optimized and scaled effectively on many-core, HPC systems.

12 pages, 9 figures

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Brief Introduction to PYTHIA 8.1
• DELPHES 3, A modular framework for fast simulation of a generic collider experiment
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Learning with Limited Numerical Precision
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets
• Training deep neural networks with low precision multiplications
• Hardware-oriented Approximation of Convolutional Neural Networks
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Fast Algorithms for Convolutional Neural Networks
• ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events
• Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs

Cited by in corpus (3)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability