PyTorch: An Imperative Style, High-Performance Deep Learning Library
arXiv:1912.01703
Abstract
Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.
12 pages, 3 figures, NeurIPS 2019
Cited by in corpus (50)
- A Simple Differentiable Programming Language
- Learning to Control PDEs with Differentiable Physics
- Permutation Invariant Graph Generation via Score-Based Generative Modeling
- Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
- Contrastive estimation reveals topic posterior information to linear models
- Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data
- DRMIME: Differentiable Mutual Information and Matrix Exponential for Multi-Resolution Image Registration
- Guiding Cascading Failure Search with Interpretable Graph Convolutional Network
- Plato Dialogue System: A Flexible Conversational AI Research Platform
- KaoKore: A Pre-modern Japanese Art Facial Expression Dataset
- Joint Distributions for TensorFlow Probability
- BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
- A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation
- XSepConv: Extremely Separated Convolution
- Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent
- Learning Convex Optimization Control Policies
- Unsupervised Denoising for Satellite Imagery using Wavelet Subband CycleGAN
- Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic
- Facial Affect Recognition in the Wild Using Multi-Task Learning Convolutional Network
- An optical diffractive deep neural network with multiple frequency-channels
- Emotion Recognition for In-the-wild Videos
- A Framework for Democratizing AI
- AdaSample: Adaptive Sampling of Hard Positives for Descriptor Learning
- Temporal Probability Calibration
- Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
- T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
- Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices
- XCloud: Design and Implementation of AI Cloud Platform with RESTful API Service
- Strength from Weakness: Fast Learning Using Weak Supervision
- CoTK: An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
- Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
- Learning by Cheating
- Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training
- Personalized acute stress classification from physiological signals with neural processes
- Deep differentiable forest with sparse attention for the tabular data
- Encoding-based Memory Modules for Recurrent Neural Networks
- Deep Self-Supervised Representation Learning for Free-Hand Sketch
- BackPACK: Packing more into backprop
- HyGCN: A GCN Accelerator with Hybrid Architecture
- Towards Graph Representation Learning in Emergent Communication
- Transfer Learning using Neural Ordinary Differential Equations
- Residual-Recursion Autoencoder for Shape Illustration Images
- Convolutional Spectral Kernel Learning
- Efficient Latent Representations using Multiple Tasks for Autonomous Driving
- A Spectral Nonlocal Block for Neural Networks
- The Benefits of Pairwise Discriminators for Adversarial Training
- Discovering Invariances in Healthcare Neural Networks
- Improve SGD Training via Aligning Mini-batches
- Scene Text Recognition with Temporal Convolutional Encoder
- Improving Visual Recognition using Ambient Sound for Supervision