Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis

Abstract

Interactive massively parallel computations are critical for machine learning and data analysis. These computations are a staple of the MIT Lincoln Laboratory Supercomputing Center (LLSC) and has required the LLSC to develop unique interactive supercomputing capabilities. Scaling interactive machine learning frameworks, such as TensorFlow, and data analysis environments, such as MATLAB/Octave, to tens of thousands of cores presents many technical challenges - in particular, rapidly dispatching many tasks through a scheduler, such as Slurm, and starting many instances of applications with thousands of dependencies. Careful tuning of launches and prepositioning of applications overcome these challenges and allow the launching of thousands of tasks in seconds on a 40,000-core supercomputer. Specifically, this work demonstrates launching 32,000 TensorFlow processes in 4 seconds and launching 262,000 Octave processes in 40 seconds. These capabilities allow researchers to rapidly explore novel machine learning architecture and data analysis algorithms.

6 pages, 7 figures, IEEE High Performance Extreme Computing Conference 2018

References in corpus (13)

• TensorFlow: A system for large-scale machine learning
• Caffe: Convolutional Architecture for Fast Feature Embedding
• The Lustre Storage Architecture
• Scalable System Scheduling for HPC and Big Data
• Computing on Masked Data: a High Performance Method for Improving Big Data Veracity
• D4M 2.0 Schema: A General Purpose High Performance Schema for the Accumulo Database
• Achieving 100,000,000 database inserts per second using Accumulo and D4M
• D4M: Bringing Associative Arrays to Database Engines
• Enabling On-Demand Database Computing with MIT SuperCloud Database Management System
• Scheduler Technologies in Support of High Performance Data Analysis
• Big Data Strategies for Data Center Infrastructure Management Using a 3D Gaming Platform
• Scalability of VM Provisioning Systems
• Benchmarking Data Analysis and Machine Learning Applications on the Intel KNL Many-Core Processor

Cited by in corpus (65)

• OSQP: An Operator Splitting Solver for Quadratic Programs
• Odd dynamics of living chiral crystals
• Transfer learning for solvation free energies: from quantum chemistry to experiments
• Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models
• Tailoring Solution Accuracy for Fast Whole-body Model Predictive Control of Legged Robots
• Residual Policy Learning
• Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)
• Dynamics, scaling behavior, and control of nuclear wrinkling
• Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning
• Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints
• Evolving Scientific Discovery by Unifying Data and Background Knowledge with AI Hilbert
• Hypersparse Neural Network Analysis of Large-Scale Internet Traffic
• New Phenomena in Large-Scale Internet Traffic
• Message Passing Networks for Molecules with Tetrahedral Chirality
• Large Scale Organization and Inference of an Imagery Dataset for Public Safety
• Spatial Temporal Analysis of 40,000,000,000,000 Internet Darkspace Packets
• Real-time Dynamics of the Schwinger Model as an Open Quantum System with Neural Density Operators
• How much physics is in a current-voltage curve? Inferring defect properties from photovoltaic device measurements
• Multi-Temporal Analysis and Scaling Relations of 100,000,000,000 Network Packets
• 75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices
• Multitask methods for predicting molecular properties from heterogeneous data
• A Green(er) World for A.I
• Learning to Recombine and Resample Data for Compositional Generalization
• Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs
• Impacts of Dynamic Line Ratings on the ERCOT Transmission System
• Optimizing Xeon Phi for Interactive Data Analysis
• Anonymized Network Sensing Graph Challenge
• Fast Mapping onto Census Blocks
• Parallel time integration using Batched BLAS (Basic Linear Algebra Subprograms) routines
• Single-shot Quantum Signal Processing Interferometry
• Maneuver Identification Challenge
• Hybrid Power-Law Models of Network Traffic
• Predicting antimicrobial activity of conjugated oligoelectrolyte molecules via machine learning
• A Stochastic Benders Decomposition Scheme for Large-Scale Stochastic Network Design
• AI Enabling Technologies: A Survey
• Scaling Graph Neural Networks to Large Proteins
• Processing of Crowdsourced Observations of Aircraft in a High Performance Computing Environment
• Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
• Best of Both Worlds: High Performance Interactive and Batch Launching
• Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices
• DBOS: A Proposal for a Data-Centric Operating System
• Self-consistent Dynamical and Chaotic Tides in the REBOUNDx framework
• Temporal Correlation of Internet Observatories and Outposts
• Cooperation and Fairness in Multi-Agent Reinforcement Learning
• Accuracy and Performance Comparison of Video Action Recognition Approaches
• On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
• Cooperative Advisory Residual Policies for Congestion Mitigation
• Technical Report: Developing a Working Data Hub
• OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User
• Uncovering the Rapidly Evolving Orbits of the Dynamic TOI-201 System
• Scaling Big Data Platform for Big Data Pipeline
• Nonlinear conjugate gradient methods: worst-case convergence rates via computer-assisted analyses
• Local retail electricity markets for distribution grid services
• DBOS Network Sensing: A Web Services Approach to Collaborative Awareness
• Equivariant Contrastive Learning
• Supercomputing Enabled Deployable Analytics for Disaster Response
• Easy Acceleration with Distributed Arrays
• An Enumerative Embedding of the Python Type System in ACL2s
• The MIT Supercloud Dataset
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
• Multi-sectoral Impacts of H2 and Synthetic Fuels Adoption for Heavy-duty Transportation Decarbonization
• HPC with Enhanced User Separation
• Decentralized Coordination of Autonomous Traffic Through Advanced Air Mobility Corridors
• Method to Characterize Potential UAS Encounters Using Open Source Data
• The Pseudo Projection Operator: Applications of Deep Learning to Projection Based Filtering in Non-Trivial Frequency Regimes