Frustrated with Replicating Claims of a Shared Model? A Solution
arXiv:1811.09737
Abstract
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and provisioning ML/DL evaluation is the main cause of this "pain point". This work discusses common pitfalls for replicating DL model evaluation, and shows that these subtle pitfalls can affect both accuracy and performance. It then proposes a solution to remedy these pitfalls called MLModelScope, a specification for repeatable model evaluation and a runtime to provision and measure experiments. We show that by easing the model specification and evaluation process, MLModelScope facilitates rapid adoption of ML/DL innovations.
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Caffe: Convolutional Architecture for Fast Feature Embedding
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Fathom: Reference Workloads for Modern Deep Learning Methods
- TensorFlow-Serving: Flexible, High-Performance ML Serving
- Benchmarking State-of-the-Art Deep Learning Software Tools
- TBD: Benchmarking and Analyzing Deep Neural Network Training
- LSTM Benchmarks for Deep Learning Frameworks
- DLPaper2Code: Auto-generation of Code from Deep Learning Research Papers
- Characterizing Co-located Datacenter Workloads: An Alibaba Case Study
Cited by in corpus (4)
- AIBench: An Industry Standard Internet Service AI Benchmark Suite
- XSP: Across-Stack Profiling and Analysis of Machine Learning Models on GPUs
- NAIS: Neural Architecture and Implementation Search and its Applications in Autonomous Driving
- AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite