6 papers
BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
Anish Saxena, Jae Hyung Ju, Hritvik Taneja +4
GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a se…
SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference
Hritvik Taneja, Anish Saxena, Abhishek Revinipati +3
The rise of reasoning models and agentic systems has made LLM token-generation latency a key bottleneck. Unlike chatbots, whose latency gains saturate at human reading speed, these…
TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout
Jae Hyung Ju, Euijun Chung, Hritvik Taneja +4
Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe…
Making Locality-aware GEMM Compatible with Page-Granularity Placement on Chiplet GPUs
Euijun Chung, Jae Hyung Ju, Hyesoon Kim
Multi-chiplet GPUs scale compute throughput and high-bandwidth memory (HBM) capacity, but their non-uniform memory system makes locality between chiplets and their data critical to…
Piccolo: Large-Scale Graph Processing with Fine-Grained In-Memory Scatter-Gather
Changmin Shin, Jaeyong Song, Hongsun Jang +7
Graph processing requires irregular, fine-grained random access patterns incompatible with contemporary off-chip memory architecture, leading to inefficient data access. This ineff…
Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection
Vima Gupta, Jae Hyung Ju, Kartik Sinha +2
Selective parameter activation provided by Mixture-of-Expert (MoE) models have made them a popular choice in modern foundational models. However, MoEs face a fundamental tension wh…