cuDNN: Efficient Primitives for Deep Learning

Abstract

We present a library of efficient implementations of deep learning primitives. Deep learning workloads are computationally intensive, and optimizing their kernels is difficult and time-consuming. As parallel architectures evolve, kernels must be reoptimized, which makes maintaining codebases difficult over time. Similar issues have long been addressed in the HPC community by libraries such as the Basic Linear Algebra Subroutines (BLAS). However, there is no analogous library for deep learning. Without such a library, researchers implementing deep learning workloads on parallel processors must create and optimize their own implementations of the main computational kernels, and this work must be repeated as new parallel processors emerge. To address this problem, we have created a library similar in intent to BLAS, with optimized routines for deep learning workloads. Our implementation contains routines for GPUs, although similarly to the BLAS library, these routines could be implemented for other platforms. The library is easy to integrate into existing frameworks, and provides optimized performance and memory usage. For example, integrating cuDNN into Caffe, a popular framework for convolutional networks, improves performance by 36% on a standard model while also reducing memory consumption.

References in corpus (5)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Fast Training of Convolutional Networks through FFTs
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs

Cited by in corpus (299)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• DeepDTA: Deep Drug-Target Binding Affinity Prediction
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• An Analysis of Deep Neural Network Models for Practical Applications
• NiftyNet: a deep-learning platform for medical imaging
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• Learning Structured Sparsity in Deep Neural Networks
• Large-Scale Optical Neural Networks based on Photoelectric Multiplication
• Convolutional Neural Networks using Logarithmic Data Representation
• Recent Advances in Convolutional Neural Networks
• Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
• Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)
• Reinforcement Learning with Augmented Data
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Deep learning of multi-element abundances from high-resolution spectroscopic data
• ECG arrhythmia classification using a 2-D convolutional neural network
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Semantic Image Segmentation via Deep Parsing Network
• wav2letter++: The Fastest Open-source Speech Recognition System
• MLPerf Training Benchmark
• Origami: A 803 GOp/s/W Convolutional Network Accelerator
• TSM: Temporal Shift Module for Efficient Video Understanding
• Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
• Deep Learning on FPGAs: Past, Present, and Future
• Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing
• Deep Learning in Mobile and Wireless Networking: A Survey
• PuRe: Robust pupil detection for real-time pervasive eye tracking
• Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Models
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Review: Deep Learning in Electron Microscopy
• FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Memory-Efficient Implementation of DenseNets
• Applying Deep Learning to Fast Radio Burst Classification
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Optimizing CNN Model Inference on CPUs
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• Benchmarking State-of-the-Art Deep Learning Software Tools
• An Electro-Photonic System for Accelerating Deep Neural Networks
• SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties
• Analysis and Optimization of Convolutional Neural Network Architectures
• Automatic identification of fossils and abiotic grains during carbonate microfacies analysis using deep convolutional neural networks
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning
• Restructuring Batch Normalization to Accelerate CNN Training
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• ZNN - A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-Core and Many-Core Shared Memory Machines
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• RepVGG: Making VGG-style ConvNets Great Again
• Automatic Large-Scale Data Acquisition via Crowdsourcing for Crosswalk Classification: A Deep Learning Approach
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
• DeLTA: GPU Performance Model for Deep Learning Applications with In-depth Memory System Traffic Analysis
• Bounding Box Regression with Uncertainty for Accurate Object Detection
• Transient-optimised real-bogus classification with Bayesian Convolutional Neural Networks -- sifting the GOTO candidate stream
• Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder
• Structured Pruning of Deep Convolutional Neural Networks
• Theano-based Large-Scale Visual Recognition with Multiple GPUs
• Pixel-wise Attentional Gating for Parsimonious Pixel Labeling
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Analyzing the Real-World Applicability of DGA Classifiers
• A Large-Scale Study on Regularization and Normalization in GANs
• Exascale Deep Learning for Scientific Inverse Problems
• StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• MegDet: A Large Mini-Batch Object Detector
• The Indirect Convolution Algorithm
• Deep Learning on Image Denoising: An overview
• AI Benchmark: Running Deep Neural Networks on Android Smartphones
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Incremental Deep Learning for Robust Object Detection in Unknown Cluttered Environments
• CosmoFlow: Using Deep Learning to Learn the Universe at Scale
• FastONN -- Python based open-source GPU implementation for Operational Neural Networks
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• A Versatile Software Systolic Execution Model for GPU Memory-Bound Kernels
• Direct Acoustics-to-Word Models for English Conversational Speech Recognition
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
• Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
• Deep Learning Markov Random Field for Semantic Segmentation
• Fast and Accurate Modeling of Transient-State Gradient-Spoiled Sequences by Recurrent Neural Networks
• HAWQV3: Dyadic Neural Network Quantization
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
• Fast ConvNets Using Group-wise Brain Damage
• Efficient Execution of Quantized Deep Learning Models: A Compiler Approach
• Identifying Flux Rope Signatures Using a Deep Neural Network
• Multi-Channel Auto-Calibration for the Atmospheric Imaging Assembly using Machine Learning
• Advances in Deep Space Exploration via Simulators & Deep Learning
• A neural network classifier for electron identification on the DAMPE experiment
• Continuum: Simple Management of Complex Continual Learning Scenarios
• Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds)
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• EvoAAA: An evolutionary methodology for automated \neural autoencoder architecture search
• FFT Convolutions are Faster than Winograd on Modern CPUs, Here is Why
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Reliability of Neural Networks Based on Spintronic Neurons
• Searching for Winograd-aware Quantized Networks
• The Effect of Silence Feature in Dimensional Speech Emotion Recognition
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Rethinking Channel Dimensions for Efficient Model Design
• Automatic Kernel Generation for Volta Tensor Cores
• Active Convolution: Learning the Shape of Convolution for Image Classification
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation
• Learning Low-Rank Approximation for CNNs
• Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
• Two-level Group Convolution
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• Deep Clustered Convolutional Kernels
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition
• DeepLens: Towards a Visual Data Management System
• Studying the potential of Graphcore IPUs for applications in Particle Physics
• cltorch: a Hardware-Agnostic Backend for the Torch Deep Neural Network Library, Based on OpenCL
• Brain Intelligence: Go Beyond Artificial Intelligence
• Automated Design Space Exploration for optimised Deployment of DNN on Arm Cortex-A CPUs
• ReCoNet: Real-time Coherent Video Style Transfer Network
• Efficient Deep Neural Networks
• Serving Recurrent Neural Networks Efficiently with a Spatial Accelerator
• Structured Convolution Matrices for Energy-efficient Deep learning
• Cortex: A Compiler for Recursive Deep Learning Models
• High-Resolution Network for Photorealistic Style Transfer
• Memory Optimization for Deep Networks
• RESPARC: A Reconfigurable and Energy-Efficient Architecture with Memristive Crossbars for Deep Spiking Neural Networks
• Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
• The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
• Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks
• Distributed Training Large-Scale Deep Architectures
• Deploying Deep Neural Networks in the Embedded Space
• Rethinking Machine Learning Development and Deployment for Edge Devices
• Smart-PGSim: Using Neural Network to Accelerate AC-OPF Power Grid Simulation
• Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
• GANDSE: Generative Adversarial Network based Design Space Exploration for Neural Network Accelerator Design
• Structured Pruning for Efficient ConvNets via Incremental Regularization
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• Rapid Exact Signal Scanning with Deep Convolutional Neural Networks
• DeepCABAC: Context-adaptive binary arithmetic coding for deep neural network compression
• Faster Neural Network Training with Approximate Tensor Operations
• Performance Evaluation of Deep Learning Tools in Docker Containers
• MDFN: Multi-Scale Deep Feature Learning Network for Object Detection
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• DeepRadioID: Real-Time Channel-Resilient Optimization of Deep Learning-based Radio Fingerprinting Algorithms
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• High-Performance Deep Learning via a Single Building Block
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Deep Tensor Convolution on Multicores
• Sparseout: Controlling Sparsity in Deep Networks
• Real-world Mapping of Gaze Fixations Using Instance Segmentation for Road Construction Safety Applications
• Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs
• BrainSlug: Transparent Acceleration of Deep Learning Through Depth-First Parallelism
• An Ultra-Efficient Memristor-Based DNN Framework with Structured Weight Pruning and Quantization Using ADMM
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Nondeterminism and Instability in Neural Network Optimization
• Demystifying the MLPerf Benchmark Suite
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Reducing the Training Time of Neural Networks by Partitioning
• dMath: A Scalable Linear Algebra and Math Library for Heterogeneous GP-GPU Architectures
• Context-Aware Single-Shot Detector
• Compiler-Level Matrix Multiplication Optimization for Deep Learning
• LazyTensor: combining eager execution with domain-specific compilers
• SMASH: Sparse Matrix Atomic Scratchpad Hashing
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
• The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding
• Array Languages Make Neural Networks Fast
• IOS: Inter-Operator Scheduler for CNN Acceleration
• Proof-of-Learning: Definitions and Practice
• A Multigrid Method for Efficiently Training Video Models
• The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
• Cross-Platform Performance Portability Using Highly Parametrized SYCL Kernels
• Factorized Bilinear Models for Image Recognition
• Quantum Optical Convolutional Neural Network: A Novel Image Recognition Framework for Quantum Computing
• LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition
• A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
• Optimal Mini-Batch Size Selection for Fast Gradient Descent
• CLAASIC: a Cortex-Inspired Hardware Accelerator
• A Unified Optimization Approach for CNN Model Inference on Integrated GPUs
• Accelerate CNN via Recursive Bayesian Pruning
• Deep Embedding Forest: Forest-based Serving with Deep Embedding Features
• Distilling with Performance Enhanced Students
• Deep learning for pedestrians: backpropagation in CNNs
• Hardware Accelerator for Adversarial Attacks on Deep Learning Neural Networks
• Energy-Aware DNN Graph Optimization
• A Metaprogramming and Autotuning Framework for Deploying Deep Learning Applications
• HALP: Hardware-Aware Latency Pruning
• Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices
• Joint Architecture and Knowledge Distillation in CNN for Chinese Text Recognition
• Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Differentiable Visual Computing
• MeDaS: An open-source platform as service to help break the walls between medicine and informatics
• Analyzing Machine Learning Workloads Using a Detailed GPU Simulator
• Predefined Sparseness in Recurrent Sequence Models
• Ancient Painting to Natural Image: A New Solution for Painting Processing
• Efficient Contextual Representation Learning Without Softmax Layer
• swCaffe: a Parallel Framework for Accelerating Deep Learning Applications on Sunway TaihuLight
• Efficient Image Evidence Analysis of CNN Classification Results
• Masked ELMo: An evolution of ELMo towards fully contextual RNN language models
• Computational Causal Inference
• Derivation and Analysis of Fast Bilinear Algorithms for Convolution
• Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method
• Model Optimization for Deep Space Exploration via Simulators and Deep Learning
• Dual-side Sparse Tensor Core
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• Fine-Grained Energy and Performance Profiling framework for Deep Convolutional Neural Networks
• Non-Blocking Simultaneous Multithreading: Embracing the Resiliency of Deep Neural Networks
• Controllable Orthogonalization in Training DNNs
• DWM: A Decomposable Winograd Method for Convolution Acceleration
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Accelerated CNN Training Through Gradient Approximation
• The Scalability for Parallel Machine Learning Training Algorithm: Dataset Matters
• Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm
• Performance evaluation of an integrated photonic convolutional neural network based on delay buffering and wavelength division multiplexing
• Post-Training Sparsity-Aware Quantization
• Performance Analysis of Deep Learning Workloads on Leading-edge Systems
• cuConv: A CUDA Implementation of Convolution for CNN Inference
• ZNNi - Maximizing the Inference Throughput of 3D Convolutional Networks on Multi-Core CPUs and GPUs
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• Post-Training BatchNorm Recalibration
• Temporal Bilinear Networks for Video Action Recognition
• Impact of GPU uncertainty on the training of predictive deep neural networks
• Mini-batch Serialization: CNN Training with Inter-layer Data Reuse
• Learning to map between ferns with differentiable binary embedding networks
• BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
• HUGE2: a Highly Untangled Generative-model Engine for Edge-computing
• NLP Workbench: Efficient and Extensible Integration of State-of-the-art Text Mining Tools
• EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM
• Enabling Highly Efficient Capsule Networks Processing Through A PIM-Based Architecture Design
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Cross-Channel Intragroup Sparsity Neural Network
• DeepFIR: Addressing the Wireless Channel Action in Physical-Layer Deep Learning
• Minimizing Area and Energy of Deep Learning Hardware Design Using Collective Low Precision and Structured Compression
• Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning
• AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks
• FUSE: Fusing STT-MRAM into GPUs to Alleviate Off-Chip Memory Access Overheads
• I/O Lower Bounds for Auto-tuning of Convolutions in CNNs
• A hybrid quantum-classical neural network with deep residual learning
• Optimizing the Whole-life Cost in End-to-end CNN Acceleration
• Homological Time Series Analysis of Sensor Signals from Power Plants
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• NeuroTrainer: An Intelligent Memory Module for Deep Learning Training
• Fast Metric Learning For Deep Neural Networks
• Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks
• Digital video microscopy enhanced by deep learning
• Creating Robust Deep Neural Networks With Coded Distributed Computing for IoT Systems
• Incremental Verification of Fixed-Point Implementations of Neural Networks
• DeepGamble: Towards unlocking real-time player intelligence using multi-layer instance segmentation and attribute detection
• Learning Equivariant Representations
• Hardware/Software Codesign for Training/Testing Multiple Neural Networks on Multiple FPGAs
• $360^o$ Surface Regression with a Hyper-Sphere Loss
• Audience Creation for Consumables -- Simple and Scalable Precision Merchandising for a Growing Marketplace
• When deep learning models on GPU can be accelerated by taking advantage of unstructured sparsity
• MERIT: Tensor Transform for Memory-Efficient Vision Processing on Parallel Architectures
• Union: A Unified HW-SW Co-Design Ecosystem in MLIR for Evaluating Tensor Operations on Spatial Accelerators
• Alleviating Bottlenecks for DNN Execution on GPUs via Opportunistic Computing
• Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations
• Accelerating convolutional neural network by exploiting sparsity on GPUs
• Binarized Convolutional Neural Networks for Efficient Inference on GPUs
• Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• A one-armed CNN for exoplanet detection from light curves
• Accelerating Deep Learning Inference with Cross-Layer Data Reuse on GPUs
• XConv: Low-memory stochastic backpropagation for convolutional layers
• Input Fast-Forwarding for Better Deep Learning
• ICA-UNet: ICA Inspired Statistical UNet for Real-time 3D Cardiac Cine MRI Segmentation
• Light curve fingerprints: an automated approach to the extraction of X-ray variability patterns with feature aggregation -- an example application to GRS 1915+105
• DeCoILFNet: Depth Concatenation and Inter-Layer Fusion based ConvNet Accelerator
• P-CapsNets: a General Form of Convolutional Neural Networks
• Content-adaptive Representation Learning for Fast Image Super-resolution
• Fast Integral Histogram Computations on GPU for Real-Time Video Analytics
• Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• MyCaffe: A Complete C# Re-Write of Caffe with Reinforcement Learning
• DSXplore: Optimizing Convolutional Neural Networks via Sliding-Channel Convolutions
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity
• Machine Learning enabled Spectrum Sharing in Dense LTE-U/Wi-Fi Coexistence Scenarios
• AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators
• CrossedWires: A Dataset of Syntactically Equivalent but Semantically Disparate Deep Learning Models
• Deep Controllable Backlight Dimming