Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

Abstract

Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware specialization to improve performance. To systematically benchmark deep learning platforms, we introduce ParaDnn, a parameterized benchmark suite for deep learning that generates end-to-end models for fully connected (FC), convolutional (CNN), and recurrent (RNN) neural networks. Along with six real-world models, we benchmark Google's Cloud TPU v2/v3, NVIDIA's V100 GPU, and an Intel Skylake CPU platform. We take a deep dive into TPU architecture, reveal its bottlenecks, and highlight valuable lessons learned for future specialized system design. We also provide a thorough comparison of the platforms and find that each has unique strengths for some types of models. Finally, we quantify the rapid performance improvements that specialized software stacks provide for the TPU and GPU platforms.

References in corpus (13)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Focal Loss for Dense Object Detection
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Scalable Methods for 8-bit Training of Neural Networks
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• Measuring the Effects of Data Parallelism on Neural Network Training
• Beyond Data and Model Parallelism for Deep Neural Networks
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• AI Matrix - Synthetic Benchmarks for DNN

Cited by in corpus (22)

• Deep learning for smart fish farming: applications, opportunities and challenges
• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
• Scientometric Review of Artificial Intelligence for Operations & Maintenance of Wind Turbines: The Past, Present and Future
• Review: Deep Learning in Electron Microscopy
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
• Accelerating Bayesian microseismic event location with deep learning
• Power-Performance Characterization of TinyML Systems
• Single Flux Quantum Based Ultrahigh Speed Spiking Neuromorphic Processor Architecture
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Hybrid Synaptic Structure for Spiking Neural Network Realization
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• Accelerating SNN Training with Stochastic Parallelizable Spiking Neurons
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• On Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat
• MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems
• EL-Attention: Memory Efficient Lossless Attention for Generation
• Towards Inference Delivery Networks: Distributing Machine Learning with Optimality Guarantees
• Optimizing High Throughput Inference on Graph Neural Networks at Shared Computing Facilities with the NVIDIA Triton Inference Server
• Towards a Domain Specific Solution for a New Generation of Wireless Modems
• Benchmarking GPUs on SVBRDF Extractor Model