180 citations · 520 across the 35 of their papers we have counts for
37 papers
SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
Gunho Park, Kyoungho Jeun, Juntaek Oh +3
Mixture-of-experts (MoE) models activate few experts per token, yet batched decoding can access nearly the entire expert pool, making expert-weight traffic a major bottleneck. Expe…
Agent-X: Full Pipeline Acceleration of On-device AI Agents
Jinha Chung, Byeongjun Shin, Jiin Kim +1
LLM-based agents deliver state-of-the-art performance across tasks but incur high end-to-end latency on edge devices. We introduce Agent-X, a software-only, accuracy-preserving fra…
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2
The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating p…
SwarmIO: Towards 100 Million IOPS SSD Emulation for Next-generation GPU-centric Storage Systems
Hyeseong Kim, Gwangoo Yeo, Minsoo Rhu
GPU-initiated I/O has emerged as a key mechanism for achieving high-throughput storage access by leveraging massive GPU thread-level parallelism, while recent industry trends point…
PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models
Eunyeong Cho, Jehyeon Bang, Ranggi Hwang +1
The emergence of reasoning-based LLMs leveraging Chain-of-Thought (CoT) inference introduces new serving challenges, as their extended reasoning phases delay user-visible output an…
ZipFlow: a Compiler-based Framework to Unleash Compressed Data Movement for Modern GPUs
Gwangoo Yeo, Zhiyang Shen, Wei Cui +5
In GPU-accelerated data analytics, the overhead of data transfer from CPU to GPU becomes a performance bottleneck when the data scales beyond GPU memory capacity due to the limited…
Mamba-X: An End-to-End Vision Mamba Accelerator for Edge Computing Devices
Dongho Yoon, Gungyu Lee, Jaewon Chang +3
Transformers have proven effective in language modeling but are limited by high computational and memory demands that grow quadratically with input sequence length. State space mod…
The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective
Jiin Kim, Byeongjun Shin, Jinha Chung +1
Large-language-model (LLM)-based AI agents have recently showcased impressive versatility by employing dynamic reasoning, an adaptive, multi-step process that coordinates with exte…
PIM-malloc: A Fast and Scalable Dynamic Memory Allocator for Processing-In-Memory (PIM) Architectures
Dongjae Lee, Bongjoon Hyun, Youngjin Kwon +1
The ability to dynamically allocate memory is fundamental in modern programming languages. However, this feature is not adequately supported in current general-purpose PIM devices.…
Debunking the CUDA Myth Towards GPU-based AI Systems
Yunjae Lee, Juntaek Lim, Jehyeon Bang +10
This paper presents a comprehensive evaluation of Intel Gaudi NPUs as an alternative to NVIDIA GPUs, which is currently the de facto standard in AI system design. First, we create…
PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers
Gwangoo Yeo, Jiin Kim, Yujeong Choi +1
NVIDIA's Multi-Instance GPU (MIG) is a feature that enables system designers to reconfigure one large GPU into multiple smaller GPU slices. This work characterizes this emerging GP…
PIM-MMU: A Memory Management Unit for Accelerating Data Transfers in Commercial PIM Systems
Dongjae Lee, Bongjoon Hyun, Taehun Kim +1
Processing-in-memory (PIM) has emerged as a promising solution for accelerating memory-intensive workloads as they provide high memory bandwidth to the processing units. This appro…
PreSto: An In-Storage Data Preprocessing System for Training Recommendation Models
Yunjae Lee, Hyeseong Kim, Minsoo Rhu
Training recommendation systems (RecSys) faces several challenges as it requires the "data preprocessing" stage to preprocess an ample amount of raw data and feed them to the GPU f…
ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models
Yujeong Choi, Jiin Kim, Minsoo Rhu
With the increasing popularity of recommendation systems (RecSys), the demand for compute resources in datacenters has surged. However, the model-wise resource allocation employed…
LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
Juntaek Lim, Youngeun Kwon, Ranggi Hwang +3
Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language pro…
vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training
Jehyeon Bang, Yujeong Choi, Myeongwoo Kim +2
As large language models (LLMs) become widespread in various application domains, a critical challenge the AI community is facing is how to train these large AI models in a cost-ef…
Pathfinding Future PIM Architectures by Demystifying a Commercial PIM Technology
Bongjoon Hyun, Taehun Kim, Dongjae Lee +1
Processing-in-memory (PIM) has been explored for decades by computer architects, yet it has never seen the light of day in real-world products due to their high design overheads an…
Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations
Yujeong Choi, John Kim, Minsoo Rhu
While providing low latency is a fundamental requirement in deploying recommendation services, achieving high resource utility is also crucial in cost-effectively maintaining the d…
GPU-based Private Information Retrieval for On-Device Machine Learning Inference
Maximilian Lam, Jeff Johnson, Wenjie Xiong +11
On-device machine learning (ML) inference can enable the use of private user data on user devices without revealing them to remote servers. However, a pure on-device solution to pr…
DiVa: An Accelerator for Differentially Private Machine Learning
Beomsik Park, Ranggi Hwang, Dongho Yoon +2
The widespread deployment of machine learning (ML) is raising serious concerns on protecting the privacy of users who contributed to the collection of training data. Differential p…
SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures
Yunjae Lee, Jinha Chung, Minsoo Rhu
Graph neural networks (GNNs) can extract features by learning both the representation of each objects (i.e., graph nodes) and the relationship across different objects (i.e., the e…
Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards
Youngeun Kwon, Minsoo Rhu
Personalized recommendation models (RecSys) are one of the most popular machine learning workload serviced by hyperscalers. A critical challenge of training RecSys is its high memo…
ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse
Jongmin Kim, Gwangho Lee, Sangpyo Kim +4
Homomorphic Encryption (HE) is one of the most promising post-quantum cryptographic schemes that enable privacy-preserving computation on servers. However, noise accumulates as we…
GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networks
Ranggi Hwang, Minhoo Kang, Jiwon Lee +3
Graph convolutional neural networks (GCNs) have emerged as a key technology in various application domains where the input data is relational. A unique property of GCNs is that its…