NVIDIA Tensor Core Programmability, Performance & Precision

Abstract

The NVIDIA Volta GPU microarchitecture introduces a specialized unit, called "Tensor Core" that performs one matrix-multiply-and-accumulate on 4x4 matrices per clock cycle. The NVIDIA Tesla V100 accelerator, featuring the Volta microarchitecture, provides 640 Tensor Cores with a theoretical peak performance of 125 Tflops/s in mixed precision. In this paper, we investigate current approaches to program NVIDIA Tensor Cores, their performances and the precision loss due to computation in mixed precision. Currently, NVIDIA provides three different ways of programming matrix-multiply-and-accumulate on Tensor Cores: the CUDA Warp Matrix Multiply Accumulate (WMMA) API, CUTLASS, a templated library based on WMMA, and cuBLAS GEMM. After experimenting with different approaches, we found that NVIDIA Tensor Cores can deliver up to 83 Tflops/s in mixed precision on a Tesla V100 GPU, seven and three times the performance in single and half precision respectively. A WMMA implementation of batched GEMM reaches a performance of 4 Tflops/s. While precision loss due to matrix multiplication with half precision input might be critical in many HPC applications, it can be considerably reduced at the cost of increased computation. Our results indicate that HPC applications using matrix multiplications can strongly benefit from using of NVIDIA Tensor Cores.

This paper has been accepted by the Eighth International Workshop on Accelerators and Hybrid Exascale Systems (AsHES) 2018

References in corpus (2)

• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems

Cited by in corpus (28)

• Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
• Accelerating Sparse Matrix-Matrix Multiplication with GPU Tensor Cores
• Characterizing Deep-Learning I/O Workloads in TensorFlow
• Three-dimensional microstructure generation using generative adversarial neural networks in the context of continuum micromechanics
• Performance Evaluation of Advanced Features in CUDA Unified Memory
• Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors
• Reference-based Defect Detection Network
• Fast Neural Representations for Direct Volume Rendering
• Recovering single precision accuracy from Tensor Cores while surpassing the FP32 theoretical peak performance
• Allo: A Programming Model for Composable Accelerator Design
• Pure Tensor Program Rewriting via Access Patterns (Representation Pearl)
• Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes
• tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads
• TensorFlow Doing HPC
• sputniPIC: an Implicit Particle-in-Cell Code for Multi-GPU Systems
• A $\mu$-mode integrator for solving evolution equations in Kronecker form
• How to keep pushing ML accelerator performance? Know your rooflines!
• AI in Space for Scientific Missions: Strategies for Minimizing Neural-Network Model Upload
• Work-Efficient Parallel Non-Maximum Suppression Kernels
• Full Version: (De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional Homomorphisms
• Multi-GPU Acceleration of the iPIC3D Implicit Particle-in-Cell Code
• Posit NPB: Assessing the Precision Improvement in HPC Scientific Applications
• Accelerating Sparse Approximate Matrix Multiplication on GPUs
• Fast Matrix Multiplication via Compiler-only Layered Data Reorganization and Intrinsic Lowering
• A compute-bound formulation of Galerkin model reduction for linear time-invariant dynamical systems
• Analyzing GPU Tensor Core Potential for Fast Reductions
• Analyzing Modern NVIDIA GPU cores
• Efficient Quantum Circuit Simulation by Tensor Network Methods on Modern GPUs