Learned Hardware/Software Co-Design of Neural Accelerators

Abstract

The use of deep learning has grown at an exponential rate, giving rise to numerous specialized hardware and software systems for deep learning. Because the design space of deep learning software stacks and hardware accelerators is diverse and vast, prior work considers software optimizations separately from hardware architectures, effectively reducing the search space. Unfortunately, this bifurcated approach means that many profitable design points are never explored. This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space. The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semi-continuous/semi-discrete. We evaluate our optimization framework by applying it to a variety of neural models, improving the energy-delay product by 18% (ResNet) and 40% (DQN) over hand-tuned state-of-the-art systems, as well as demonstrating strong results on other neural network architectures, such as MLPs and Transformers.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Playing Atari with Deep Reinforcement Learning
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• Measuring the Algorithmic Efficiency of Neural Networks

Cited by in corpus (4)

• A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators
• Constrained Discrete Black-Box Optimization using Mixed-Integer Programming
• Data-Driven Offline Optimization For Architecting Hardware Accelerators
• Enabling Cross-Domain Communication: How to Bridge the Gap between AI and HW Engineers