Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computing

Abstract

In-Memory Computing (IMC) platforms such as analog crossbars are gaining focus as they facilitate the acceleration of low-precision Deep Neural Networks (DNNs) with high area- & compute-efficiencies. However, the intrinsic non-idealities in crossbars, which are often non-deterministic and non-linear, degrade the performance of the deployed DNNs. In addition to quantization errors, most frequently encountered non-idealities during inference include crossbar circuit-level parasitic resistances and device-level non-idealities such as stochastic read noise and temporal drift. In this work, our goal is to closely examine the distortions caused by these non-idealities on the dot-product operations in analog crossbars and explore the feasibility of a nearly training-less solution via crossbar-aware fine-tuning of batchnorm parameters in real-time to mitigate the impact of the non-idealities. This enables reduction in hardware costs in terms of memory and training energy for IMC noise-aware retraining of the DNN weights on crossbars.

arXiv commentAccepted in Great Lakes Symposium on VLSI 2023 (GLSVLSI 2023) conference

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Accurate deep neural network inference using computational phase-change memory
• Multiscale Co-Design Analysis of Energy, Latency, Area, and Accuracy of a ReRAM Analog Neural Training Accelerator
• Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbars
• Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization
• NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems

Cited by in corpus (4)

• Fast offset corrected in-memory training
• Enabling Energy-Efficient Deployment of Large Language Models on Memristor Crossbar: A Synergy of Large and Small
• ClipFormer: Key-Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation
• ThRIve: Thermally Robust CNN Inference via Low-Rank Adaptation in Heterogeneous PIM Architectures