CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework

Abstract

Recently, automated co-design of machine learning (ML) models and accelerator architectures has attracted significant attention from both the industry and academia. However, most co-design frameworks either explore a limited search space or employ suboptimal exploration techniques for simultaneous design decision investigations of the ML model and the accelerator. Furthermore, training the ML model and simulating the accelerator performance is computationally expensive. To address these limitations, this work proposes a novel neural architecture and hardware accelerator co-design framework, called CODEBench. It is composed of two new benchmarking sub-frameworks, CNNBench and AccelBench, which explore expanded design spaces of convolutional neural networks (CNNs) and CNN accelerators. CNNBench leverages an advanced search technique, BOSHNAS, to efficiently train a neural heteroscedastic surrogate model to converge to an optimal CNN architecture by employing second-order gradients. AccelBench performs cycle-accurate simulations for a diverse set of accelerator architectures in a vast design space. With the proposed co-design method, called BOSHCODE, our best CNN-accelerator pair achieves 1.4% higher accuracy on the CIFAR-10 dataset compared to the state-of-the-art pair, while enabling 59.1% lower latency and 60.8% lower energy consumption. On the ImageNet dataset, it achieves 3.7% higher Top1 accuracy at 43.8% lower latency and 11.2% lower energy consumption. CODEBench outperforms the state-of-the-art framework, i.e., Auto-NBA, by achieving 1.5% higher accuracy and 34.7x higher throughput, while enabling 11.0x lower energy-delay product (EDP) and 4.0x lower chip area on CIFAR-10.

arXiv commentPublished at ACM Transactions on Embedded Computing Systems. Code available at https://github.com/jha-lab/codebench

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• Neural Architecture Search: A Survey
• Deep Learning with Limited Numerical Precision
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• MCUNet: Tiny Deep Learning on IoT Devices
• COSCO: Container Orchestration using Co-Simulation and Gradient Based Optimization for Fog Computing Environments
• Image Classification at Supercomputer Scale
• D-VAE: A Variational Autoencoder for Directed Acyclic Graphs
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
• Software-defined Design Space Exploration for an Efficient DNN Accelerator Architecture
• FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?
• Fast Design Space Exploration of Nonlinear Systems: Part I

Cited by in corpus (2)

• MetaML-Pro: Cross-Stage Design Flow Automation for Efficient Deep Learning Acceleration
• Systolic Arrays and Structured Pruning Co-design for Efficient Transformers in Edge Systems