TransAxx: Efficient Transformers with Approximate Computing

Abstract

Vision Transformer (ViT) models which were recently introduced by the transformer architecture have shown to be very competitive and often become a popular alternative to Convolutional Neural Networks (CNNs). However, the high computational requirements of these models limit their practical applicability especially on low-power devices. Current state-of-the-art employs approximate multipliers to address the highly increased compute demands of DNN accelerators but no prior research has explored their use on ViT models. In this work we propose TransAxx, a framework based on the popular PyTorch library that enables fast inherent support for approximate arithmetic to seamlessly evaluate the impact of approximate computing on DNNs such as ViT models. Using TransAxx we analyze the sensitivity of transformer models on the ImageNet dataset to approximate multiplications and perform approximate-aware finetuning to regain accuracy. Furthermore, we propose a methodology to generate approximate accelerators for ViT models. Our approach uses a Monte Carlo Tree Search (MCTS) algorithm to efficiently search the space of possible configurations using a hardware-driven hand-crafted policy. Our evaluation demonstrates the efficacy of our methodology in achieving significant trade-offs between accuracy and power, resulting in substantial gains without compromising on performance.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining
• Towards Accurate Post-Training Quantization for Vision Transformer
• AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch
• TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU
• AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference
• Automated Circuit Approximation Method Driven by Data Distribution
• Automated Design Approximation to Overcome Circuit Aging
• Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search
• An Energy-Efficient Generic Accuracy Configurable Multiplier Based on Block-Level Voltage Overscaling