9 papers
CascadeEP: Asynchronous Expert Execution for MoE Prefill under Attention Imbalance
Jin Qin, Tiancheng Hu, Shiyan Wang +12
Mixture-of-experts (MoE) serving commonly deploys data and expert parallelism (DEP): attention replicas run distinct request batches while routed experts are sharded across an expe…
Can Large Language Models Recover Semantic Optimization Opportunities That Compilers Miss?
Hailong Jiang, Feng Yu, Emran Hossain +4
Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation. We ask whether large language models (LLMs)…
BMOA: Baseline-Mechanism-Outcome Attribution for Compiler-Induced Numerical Deviations
Hailong Jiang, Emran Hossain, Feng Yu +4
Formalizing compiler-aware numerical correctness requires distinguishing what an observed floating-point difference means, what compiler behavior the evidence supports, and what nu…
Decode-Time Grammars: Constrained LLM Generation over a Refinement Order of Grammar Fragments
Shuoming Zhang, Ruiyuan Xu, Haofeng Li +7
Large language models now write a growing share of the world's code, increasingly inside agents and serving systems that compile, execute, or dispatch generated code without line-b…
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Tiancheng Hu, Jin Qin, Zheng Wang +10
Disaggregation maps parts of an AI workload to different types of GPUs, offering a path to utilize modern heterogeneous GPU clusters. However, existing solutions operate at a coars…
The New Compiler Stack: A Survey on the Synergy of LLMs and Compilers
Shuoming Zhang, Jiacheng Zhao, Qiuchu Yu +4
This survey has provided a systematic overview of the emerging field of LLM-enabled compilation by addressing several key research questions. We first answered how LLMs are being i…
KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference
Huawei Zhang, Chunwei Xia, Zheng Wang
Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long…
LEGO-Compiler: Enhancing Neural Compilation Through Translation Composability
Shuoming Zhang, Jiacheng Zhao, Chunwei Xia +4
Large language models (LLMs) have the potential to revolutionize how we design and implement compilers and code translation tools. However, existing LLMs struggle to handle long an…
When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output
Shuoming Zhang, Jiacheng Zhao, Hanyuan Dong +9
Content Warning: This paper may contain unsafe or harmful content generated by LLMs that may be offensive to readers. Large Language Models (LLMs) increasingly serve as tooling pla…