9 papers
REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
Qian Zhang, Yaoming Li, Zhewen Tan +9
Post-training quantization (PTQ) is essential for deploying large language models (LLMs) under strict resource constraints. State-of-the-art PTQ methods quantize each layer with a…
Adaptive Context Parallelism for Production LLM Serving
Jiarui Guo, Rongle Wang, Peijun Huang +7
As LLM context windows expand and input sequences grow longer, serving systems face increasing computational and memory demands. Context parallelism (CP), which partitions the inpu…
ArborMem: Navigating Interaction States with Memory Forests
Zongwei Lv, Yuemeng Xu, Yilun Yao +8
Large language models increasingly serve as persistent conversational assistants, requiring memory that preserves relevant experience and maintains continuity across interactions.…
InSituANN: Revisiting IVF for PCIe-Efficient Billion-Scale Vector Search
Yuemeng Xu, Zongxi Liu, Junyu Long +7
Approximate nearest neighbor search (ANNS) over billion-scale vector datasets has become a foundational operator for modern retrieval systems, powering large-scale recommendation,…
RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Zongwei Lv, Yaoming Li, Zhewen Tan +7
Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introd…
RTP-LLM: High-Performance Alibaba LLM Inference Engine
Boyu Tan, Jiarui Guo, Zongwei Lv +26
Large Language Models (LLMs) have revolutionized AI applications, but deploying them at scale presents significant challenges. We present RTP-LLM, a high-performance inference engi…
Rethinking Retrieval-Augmentation as Synthesis: A Query-Aware Context Merging Approach
Jiarui Guo, Yuemeng Xu, Zongwei Lv +6
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to extend their existing knowledge by dynamically incorporating external information. However, practical d…
HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs
Guoan Wang, Feiyu Wang, Zongwei Lv +2
As large language models (LLMs) continue to scale, deployment is increasingly bottlenecked by the memory wall, motivating a shift toward extremely low-bit quantization. However, mo…
LLM-Sketch: Enhancing Network Sketches with LLM
Yuanpeng Li, Zhen Xu, Zongwei Lv +3
Network stream mining is fundamental to many network operations. Sketches, as compact data structures that offer low memory overhead with bounded accuracy, have emerged as a promis…