Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

Abstract

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. The majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Accepted for publication in the FCCM'24 Journal Track and will appear in ACM Transactions on Reconfigurable Technology and Systems (TRETS)

References in corpus (17)

• LLaMA: Open and Efficient Foundation Language Models
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• NPE: An FPGA-based Overlay Processor for Natural Language Processing
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
• Reducing Activation Recomputation in Large Transformer Models
• A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining
• Accelerating Large Language Model Decoding with Speculative Sampling
• Full Stack Optimization of Transformer Inference: a Survey
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
• RPTQ: Reorder-based Post-training Quantization for Large Language Models

Cited by in corpus (4)

• Allo: A Programming Model for Composable Accelerator Design
• Reconfigurable Stream Network Architecture
• StreamTensor: Make Tensors Stream in Dataflow Accelerators for LLMs
• A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs