Accurate deep neural network inference using computational phase-change memory

Abstract

In-memory computing is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. Crossbar arrays of resistive memory devices can be used to encode the network weights and perform efficient analog matrix-vector multiplications without intermediate movements of data. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to in-memory computing hardware based on phase-change memory (PCM). We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on the CIFAR-10 dataset and a top-1 accuracy on the ImageNet benchmark of 71.6% after mapping the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one day period, where each of the 361,722 synaptic weights of the network is programmed on just two PCM devices organized in a differential configuration.

This is a pre-print of an article accepted for publication in Nature Communications

References in corpus (6)

• Neuromorphic computing with multi-memristive synapses
• Mixed-Precision In-Memory Computing
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Mixed-precision deep learning based on computational memory
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• 5 Parallel Prism: A topology for pipelined implementations of convolutional neural networks using computational memory

Cited by in corpus (42)

• Parallel convolution processing using an integrated photonic tensor core
• A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrays
• Mixed-precision deep learning based on computational memory
• Robust High-dimensional Memory-augmented Neural Networks
• Reconfigurable Compute-In-Memory on Field-Programmable Ferroelectric Diodes
• Applications and Techniques for Fast Machine Learning in Science
• On the Accuracy of Analog Neural Network Inference Accelerators
• Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference
• Bulk-Switching Memristor-based Compute-In-Memory Module for Deep Neural Network Training
• Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• Stateful Logic using Phase Change Memory
• Process, Bias and Temperature Scalable CMOS Analog Computing Circuits for Machine Learning
• Demonstrating Analog Inference on the BrainScaleS-2 Mobile System
• Denoising Noisy Neural Networks: A Bayesian Approach with Compensation
• Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics
• All-in-One Analog AI Hardware: On-Chip Training and Inference with Conductive-Metal-Oxide/HfOx ReRAM Devices
• ALPINE: Analog In-Memory Acceleration with Tight Processor Integration for Deep Learning
• The Inherent Adversarial Robustness of Analog In-Memory Computing
• Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computing
• Improving the Accuracy of Analog-Based In-Memory Computing Accelerators Post-Training
• Xpikeformer: Hybrid Analog-Digital Hardware Acceleration for Spiking Transformers
• CiMBA: Accelerating Genome Sequencing through On-Device Basecalling via Compute-in-Memory
• Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticity
• WHYPE: A Scale-Out Architecture with Wireless Over-the-Air Majority for Scalable In-memory Hyperdimensional Computing
• CRAFT: Criticality-Aware Fault-Tolerance Enhancement Techniques for Emerging Memories-Based Deep Neural Networks
• End-to-end 100-TOPS/W Inference With Analog In-Memory Computing: Are We There Yet?
• Approximations in Deep Learning
• Bayes2IMC: In-Memory Computing for Bayesian Binary Neural Networks
• Effect of memristorś potentiation-depression curves peculiarities in the convergence of physical perceptrons
• Multi-mem behavior at reduced voltages in La${}_{1/2}$Sr${}_{1/2}$Mn${}_{1/2}$Co${}_{1/2}$O${}_{3-x}$ perovskite modified with Sm:CeO${}_2$
• Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions
• Memristive, Spintronic, and 2D-Materials-Based Devices to Improve and Complement Computing Hardware
• 2022 Roadmap on Neuromorphic Computing and Engineering
• ThRIve: Thermally Robust CNN Inference via Low-Rank Adaptation in Heterogeneous PIM Architectures
• Characteristic time of transition from write error to retention error in voltage-controlled magnetoresistive random-access memory
• Digital-analog concept for superconducting perceptron-like neural networks
• Efficient transformer adaptation for analog in-memory computing via low-rank adapters
• Nonideality-Aware Training for Accurate and Robust Low-Power Memristive Neural Networks
• Voltage Controlled Energy Efficient Domain Wall Synapses with Stochastic Distribution of Quantized Weights in the Presence of Thermal Noise and Edge Roughness