PolyLUT: Learning Piecewise Polynomials for Ultra-Low Latency FPGA LUT-based Inference

Abstract

Field-programmable gate arrays (FPGAs) are widely used to implement deep learning inference. Standard deep neural network inference involves the computation of interleaved linear maps and nonlinear activation functions. Prior work for ultra-low latency implementations has hardcoded the combination of linear maps and nonlinear activations inside FPGA lookup tables (LUTs). Our work is motivated by the idea that the LUTs in an FPGA can be used to implement a much greater variety of functions than this. In this paper, we propose a novel approach to training neural networks for FPGA deployment using multivariate polynomials as the basic building block. Our method takes advantage of the flexibility offered by the soft logic, hiding the polynomial evaluation inside the LUTs with minimal overhead. We show that by using polynomial building blocks, we can achieve the same accuracy using considerably fewer layers of soft logic than by using linear functions, leading to significant latency and area improvements. We demonstrate the effectiveness of this approach in three tasks: network intrusion detection, jet identification at the CERN Large Hadron Collider, and handwritten digit recognition using the MNIST dataset.

References in corpus (5)

• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• Fast inference of deep neural networks in FPGAs for particle physics
• Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
• Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
• Fast inference of Boosted Decision Trees in FPGAs for particle physics

Cited by in corpus (6)

• TreeLUT: An Efficient Alternative to Deep Neural Networks for Inference Acceleration Using Gradient Boosted Decision Trees
• NeuraLUT: Hiding Neural Network Density in Boolean Synthesizable Functions
• Field-Programmable Gate Array Architecture for Deep Learning: Survey & Future Directions
• ReducedLUT: Table Decomposition with "Don't Care" Conditions
• Table-Lookup MAC: Scalable Processing of Quantised Neural Networks in FPGA Soft Logic
• LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neurons