4 papers
Accelerating Data Preprocessing for Efficient Vision Model Inference on Jetson Edge Device
Tian Chen, Nawras Alnaasan, Jinghan Yao +4
Data preprocessing is a crucial part of deep learning workflows on edge devices. However, decoding data saved in JPEG format is very compute-intensive and occupies a major portion…
Characterizing Communication Patterns in Distributed Large Language Model Inference
Lang Xu, Kaushik Kandadi Suresh, Quentin Anthony +2
Large Language Models (LLMs) built on transformer architectures have transformed natural language processing, achieving remarkable performance across diverse applications. While di…
Accelerating Large Language Model Training with Hybrid GPU-based Compression
Lang Xu, Quentin Anthony, Qinghua Zhou +5
Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) traini…
Flover: A Temporal Fusion Framework for Efficient Autoregressive Model Parallel Inference
Jinghan Yao, Nawras Alnaasan, Tian Chen +4
Autoregressive models, despite their commendable performance in a myriad of generative tasks, face challenges stemming from their inherently sequential structure. Inference on thes…