Tune: A Research Platform for Distributed Model Selection and Training
arXiv:1807.05118
Abstract
Modern machine learning algorithms are increasingly computationally demanding, requiring specialized hardware and distributed computation to achieve high performance in a reasonable time frame. Many hyperparameter search algorithms have been proposed for improving the efficiency of model selection, however their adaptation to the distributed compute environment is often ad-hoc. We propose Tune, a unified framework for model selection and training that provides a narrow-waist interface between training scripts and search algorithms. We show that this interface meets the requirements for a broad range of hyperparameter search algorithms, allows straightforward scaling of search to large clusters, and simplifies algorithm implementation. We demonstrate the implementation of several state-of-the-art hyperparameter search algorithms in Tune. Tune is available at http://ray.readthedocs.io/en/latest/tune.html.
8 Pages, Presented at the 2018 ICML AutoML workshop
References in corpus (4)
Cited by in corpus (108)
- Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
- Hyper-Parameter Optimization: A Review of Algorithms and Applications
- Self-attention for raw optical Satellite Time Series Classification
- Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review
- Infusing Theory into Deep Learning for Interpretable Reactivity Prediction
- Rethinking CNN Models for Audio Classification
- ViscNet: Neural network for predicting the fragility index and the temperature-dependency of viscosity
- Deep learning-based surrogate model for 3-D patient-specific computational fluid dynamics
- A System for Massively Parallel Hyperparameter Tuning
- Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing
- Quantum Reinforcement Learning: the Maze problem
- AlphaClean: Automatic Generation of Data Cleaning Pipelines
- A Theoretical Framework for Target Propagation
- HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks
- Thermal experiments for fractured rock characterization: theoretical analysis and inverse modeling
- A Physics-Enforced Neural Network to Predict Polymer Melt Viscosity
- VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
- MIST-CF: Chemical formula inference from tandem mass spectra
- Sustainability of Data Center Digital Twins with Reinforcement Learning
- Leveraging Variational Autoencoders for Parameterized MMSE Estimation
- ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Markets
- Node Classification on Graphs with Few-Shot Novel Labels via Meta Transformed Network Embedding
- FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
- c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
- Evaluating Generic Auto-ML Tools for Computational Pathology
- Model-based Asynchronous Hyperparameter and Neural Architecture Search
- Show Your Work: Improved Reporting of Experimental Results
- A stacked deep convolutional neural network to predict the remaining useful life of a turbofan engine
- Learning 3D Representations of Molecular Chirality with Invariance to Bond Rotations
- A machine learning and feature engineering approach for the prediction of the uncontrolled re-entry of space objects
- Probabilistic Neural Data Fusion for Learning from an Arbitrary Number of Multi-fidelity Data Sets
- Training Agents using Upside-Down Reinforcement Learning
- Deep learning enhanced noise spectroscopy of a spin qubit environment
- Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning
- Machine learning modeling of the atomic structure and physical properties of alkali and alkaline-earth aluminosilicate glasses and melts
- Beltrami Flow and Neural Diffusion on Graphs
- Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines
- Standardized Non-Intrusive Reduced Order Modeling Using Different Regression Models With Application to Complex Flow Problems
- Physics-Based Hybrid Machine Learning for Critical Heat Flux Prediction with Uncertainty Quantification
- Interpretable multiscale Machine Learning-Based Parameterizations of Convection for ICON
- Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
- Machine learning classification of non-Markovian noise disturbing quantum dynamics
- Differentiating Viral and Bacterial Infections: A Machine Learning Model Based on Routine Blood Test Values
- Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections
- Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding
- Deep learning with plasma plume image sequences for anomaly detection and prediction of growth kinetics during pulsed laser deposition
- Optimizing a Digital Twin for Fault Diagnosis in Grid Connected Inverters -- A Bayesian Approach
- AutoPV: Automated photovoltaic forecasts with limited information using an ensemble of pre-trained models
- Model-Parallel Model Selection for Deep Learning Systems
- How Useful is Self-Supervised Pretraining for Visual Tasks?
- Optimal foraging strategies can be learned
- Prediction of soft proton intensities in the near-Earth space using machine learning
- CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms
- Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
- Near real-time streaming analysis of big fusion data
- Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data
- A Framework for Democratizing AI
- Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum
- SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning
- Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
- An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models
- Democratizing Production-Scale Distributed Deep Learning
- Numerically Solving Parametric Families of High-Dimensional Kolmogorov Partial Differential Equations via Deep Learning
- Generalized Latency Performance Estimation for Once-For-All Neural Architecture Search
- PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
- Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty
- Distributed Training and Optimization Of Neural Networks
- Inferring Javascript types using Graph Neural Networks
- A Bayesian Generative Adversarial Network (GAN) to Generate Synthetic Time-Series Data, Application in Combined Sewer Flow Prediction
- CNN-based local features for navigation near an asteroid
- Stable Invariant Models via Koopman Spectra
- Reinforcement Learning reveals fundamental limits on the mixing of active particles
- A contrastive rule for meta-learning
- Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning
- Stage-based Hyper-parameter Optimization for Deep Learning
- A Scalable and Cloud-Native Hyperparameter Tuning System
- Hippo: Taming Hyper-parameter Optimization of Deep Learning with Stage Trees
- MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields
- Review and Examination of Input Feature Preparation Methods and Machine Learning Models for Turbulence Modeling
- Using Non-Linear Causal Models to Study Aerosol-Cloud Interactions in the Southeast Pacific
- On the Accuracy of CRNNs for Line-Based OCR: A Multi-Parameter Evaluation
- Estimation of Counterfactual Interventions under Uncertainties
- Understanding and Optimizing Packed Neural Network Training for Hyper-Parameter Tuning
- SplitVAEs: Decentralized scenario generation from siloed data for stochastic optimization problems
- Improving Fast Minimum-Norm Attacks with Hyperparameter Optimization
- Proximal Mapping for Deep Regularization
- HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
- Model Order Selection with Variational Autoencoding
- Running Alchemist on Cray XC and CS Series Supercomputers: Dask and PySpark Interfaces, Deployment Options, and Data Transfer Times
- Training Image Selection using Recurrent Neural Networks: An Application in Hydrogeology
- Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems
- Understanding the factors driving the opioid epidemic using machine learning
- On the Generalization of Agricultural Drought Classification from Climate Data
- Bidirectional Long Short-Term Memory (BLSTM) neural networks for reconstruction of top-quark pair decay kinematics
- Explore BiLSTM-CRF-Based Models for Open Relation Extraction
- High Quality Related Search Query Suggestions using Deep Reinforcement Learning
- Paramater Optimization for Manipulator Motion Planning using a Novel Benchmark Set
- Reinforcement Learning with Neural Networks for Quantum Multiple Hypothesis Testing
- PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters
- AMU-EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERT
- Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts
- TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification
- Classifying Diagrams and Their Parts using Graph Neural Networks: A Comparison of Crowd-Sourced and Expert Annotations
- Matching with Transformers in MELT
- Determining Individual Origin Similarity (DInOS): Binary Classification of Authors Using Stylometric Features
- CrossedWires: A Dataset of Syntactically Equivalent but Semantically Disparate Deep Learning Models
- Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses
- Genealogical Population-Based Training for Hyperparameter Optimization