16 papers
WaferTrans: Enabling IOMMU-free Distributed Virtual Address Translation for Wafer-scale GPUs
Xinru Tang, Jingxiang Hou, Guanghong Wu +2
Wafer-scale GPUs (WSGs) provide sufficient on-wafer bandwidth to make near-lossless Unified Memory feasible. However, existing designs still rely on a CPU-IOMMU to translate remote…
MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference
Zichuan Wang, Huizheng Wang, Yuheng Xiao +6
Large language models (LLMs) are increasingly used in prefill-only workloads, where end-to-end latency is dominated by the prefill phase. For long-context prefill, communication ov…
FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning
Takanori Yoshimoto, Yang Hu, Naruya Kondo +1
Latent actions provide a compact interface between action-free video and downstream decision-making, yet existing Latent Action Models (LAMs) force every transition through a fixed…
Beyond Similarity: Trustworthy Memory Search for Personal AI Agents
Jiawen Zhang, Kejia Chen, Jiachen Ma +7
Personal AI agents increasingly rely on long-term memory to provide persistent personalization across sessions. However, existing memory pipelines are largely driven by semantic si…
PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
Huizheng Wang, Hongbin Wang, Zichuan Wang +5
Attention-based models have revolutionized AI, but the quadratic cost of self-attention incurs severe computational and memory overhead. Sparse attention methods alleviate this by…
Designing Spatial Architectures for Sparse Attention: STAR Accelerator via Cross-Stage Tiling
Huizheng Wang, Taiquan Wei, Hongbin Wang +6
Large language models (LLMs) rely on self-attention for contextual understanding, demanding high-throughput inference and large-scale token parallelism (LTPP). Existing dynamic spa…
cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution
Jinwu Chen, Qidie Wu, Bin Li +5
Optimizing CUDA kernels is a challenging and labor-intensive task, given the need for hardware-software co-design expertise and the proprietary nature of high-performance kernel li…
TEMP: A Memory Efficient Physical-aware Tensor Partition-Mapping Framework on Wafer-scale Chips
Huizheng Wang, Taiquan Wei, Zichuan Wang +8
Large language models (LLMs) demand significant memory and computation resources. Wafer-scale chips (WSCs) provide high computation power and die-to-die (D2D) bandwidth but face a…
WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chip
Huizheng Wang, Zichuan Wang, Hongbin Wang +5
Training large language models (LLMs) imposes extreme demands on computation, memory capacity, and interconnect bandwidth, driven by their ever-increasing parameter scales and inte…
BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination
Huizheng Wang, Hongbin Wang, Shaojun Wei +2
Attention-based large language models (LLMs) have transformed modern AI applications, but the quadratic cost of self-attention imposes significant compute and memory overhead. Dyna…
LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Model
Huizheng Wang, Hongbin Wang, Shaojun Wei +2
Attention-based Transformers have revolutionized natural language processing (NLP) and shown strong performance in computer vision (CV) tasks. However, as the input sequence varies…
MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference
Xinru Tang, Jingxiang Hou, Dingcheng Jiang +9
As large language models (LLMs) continue to scale up, mixture-of-experts (MoE) has become a common technology in SOTA models. MoE models rely on expert parallelism (EP) to alleviat…
MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
Huizheng Wang, Zichuan Wang, Zhiheng Yue +8
Large language models (LLMs) face significant inference latency due to inefficiencies in GEMM operations, weight access, and KV cache access, especially in real-time scenarios. Thi…
Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programs
Xiao Zhang, Yaoyao Ding, Bolin Sun +3
Efficient GPU programming is crucial for achieving high performance in deep learning (DL) applications. The performance of GPU programs depends on how data is parallelized across t…
Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
Mingcong Song, Xinru Tang, Fengfan Hou +9
Meeting growing demands for low latency and cost efficiency in production-grade large language model (LLM) serving systems requires integrating advanced optimization techniques. Ho…
Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
WenZheng Zhang, Yang Hu, Jing Shi +1
Scaling Deep Neural Networks (DNNs) requires significant computational resources in terms of GPU quantity and compute capacity. In practice, there usually exists a large number of…