Fathom: Reference Workloads for Modern Deep Learning Methods

Abstract

Deep learning has been popularized by its recent successes on challenging artificial intelligence problems. One of the reasons for its dominance is also an ongoing challenge: the need for immense amounts of computational power. Hardware architects have responded by proposing a wide array of promising ideas, but to date, the majority of the work has focused on specific algorithms in somewhat narrow application domains. While their specificity does not diminish these approaches, there is a clear need for more flexible solutions. We believe the first step is to examine the characteristics of cutting edge models from across the deep learning community. Consequently, we have assembled Fathom: a collection of eight archetypal deep learning workloads for study. Each of these models comes from a seminal work in the deep learning community, ranging from the familiar deep convolutional neural network of Krizhevsky et al., to the more exotic memory networks from Facebook's AI research group. Fathom has been released online, and this paper focuses on understanding the fundamental performance characteristics of each model. We use a set of application-level modeling tools built around the TensorFlow deep learning framework in order to analyze the behavior of the Fathom workloads. We present a breakdown of where time is spent, the similarities between the performance profiles of our models, an analysis of behavior in inference and training, and the effects of parallelism on scaling.

Proceedings of the IEEE International Symposium on Workload Characterization, 2016

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• cuDNN: Efficient Primitives for Deep Learning
• End-To-End Memory Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network

Cited by in corpus (42)

• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• MLPerf Training Benchmark
• Neurostream: Scalable and Energy Efficient Deep Learning with Smart Memory Cubes
• The unreasonable effectiveness of the forget gate
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
• Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning
• I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation
• Hierarchical Training of Deep Neural Networks Using Early Exiting
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Predicting the Computational Cost of Deep Learning Models
• AI Matrix: A Deep Learning Benchmark for Alibaba Data Centers
• Tango: A Deep Neural Network Benchmark Suite for Various Accelerators
• InferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking System
• Frustrated with Replicating Claims of a Shared Model? A Solution
• RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing
• Demystifying the MLPerf Benchmark Suite
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• DeepOBS: A Deep Learning Optimizer Benchmark Suite
• A Workload and Programming Ease Driven Perspective of Processing-in-Memory
• Characterizing Deep Learning Training Workloads on Alibaba-PAI
• QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware
• A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• AIBench Scenario: Scenario-distilling AI Benchmarking
• Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
• MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems
• Demystifying BERT: Implications for Accelerator Design
• Performance Analysis of Deep Learning Workloads on Leading-edge Systems
• Fine-Grained Energy and Performance Profiling framework for Deep Convolutional Neural Networks
• HPC AI500: A Benchmark Suite for HPC AI Systems
• Data Motif-based Proxy Benchmarks for Big Data and AI Workloads
• Energy Predictive Models for Convolutional Neural Networks on Mobile Platforms
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• AIPerf: Automated machine learning as an AI-HPC benchmark
• Bosch Deep Learning Hardware Benchmark
• DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended)