Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision

Abstract

Low-precision training has become a popular approach to reduce compute requirements, memory footprint, and energy consumption in supervised learning. In contrast, this promising approach has not yet enjoyed similarly widespread adoption within the reinforcement learning (RL) community, partly because RL agents can be notoriously hard to train even in full precision. In this paper we consider continuous control with the state-of-the-art SAC agent and demonstrate that a naïve adaptation of low-precision methods from supervised learning fails. We propose a set of six modifications, all straightforward to implement, that leaves the underlying agent and its hyperparameters unchanged but improves the numerical stability dramatically. The resulting modified SAC agent has lower memory and compute requirements while matching full-precision rewards, demonstrating that low-precision training can substantially accelerate state-of-the-art RL without parameter tuning.

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Dota 2 with Large Scale Deep Reinforcement Learning
• Mixed Precision Training
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Fixed Point Quantization of Deep Convolutional Networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Learning Latent Dynamics for Planning from Pixels
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• A Study of BFLOAT16 for Deep Learning Training
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• TorchBeast: A PyTorch Platform for Distributed RL
• SWALP : Stochastic Weight Averaging in Low-Precision Training
• Evaluating the Performance of Reinforcement Learning Algorithms
• Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning
• Accelerating Reinforcement Learning through GPU Atari Emulation
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• High-Throughput Synchronous Deep RL
• Revisiting BFloat16 Training
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Design