1 citations · 1 across the 10 of their papers we have counts for
11 papers
CoRun: Padding is Simple and Efficient for Deterministic LLM Inference
Shiju Zhao, Jiacheng Yang, Qihang Chen +4
Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation a…
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
Yipeng Liu, Chang Liu, Si Shen +16
The deployment of Mixture-of-Experts (MoE) models on production high-bandwidth superpods, such as NVIDIA's NVL72/576 and Huawei's CloudMatrix384, introduces critical challenges bey…
An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG
Chenxin Mao, Shangyu Liu, Zhenzhe Zheng +3
Retrieval-Augmented Generation (RAG) empowers LLMs with external knowledge, making cross-institutional domain-specific knowledge base integration a highly promising deployment para…
Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequences
Chen Gong, Zhenzhe Zheng, Yiliu Chen +3
Machine learning models are widely integrated into modern mobile apps to analyze user behaviors and deliver personalized services. Ensuring low-latency on-device model execution is…
AnyPro: Preference-Preserving Anycast Optimization based on Strategic AS-Path Prepending
Minyuan Zhou, Yuning Chen, Jiaqi Zheng +11
Operating large-scale anycast networks is challenging because client-to-site mappings often misalign with operator's expectation due to opaque inter-domain routing. We present AnyP…
You Need an Encoder for Native Position-Independent Caching
Shiju Zhao, Junhao Hu, Jiaqi Zheng +1
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Ca…
Bi-Level Decision-Focused Causal Learning for Large-Scale Marketing Optimization: Bridging Observational and Experimental Data
Shuli Zhang, Hao Zhou, Jiaqi Zheng +4
Online Internet platforms require sophisticated marketing strategies to optimize user retention and platform revenue -- a classical resource allocation problem. Traditional solutio…
MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving
Shiju Zhao, Junhao Hu, Rongxiao Huang +2
The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merel…
CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
Hongxuan Zhang, Yao Zhao, Jiaqi Zheng +3
The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linea…
Reinfier and Reintrainer: Verification and Interpretation-Driven Safe Deep Reinforcement Learning Frameworks
Zixuan Yang, Jiaqi Zheng, Guihai Chen
Ensuring verifiable and interpretable safety of deep reinforcement learning (DRL) is crucial for its deployment in real-world applications. Existing approaches like verification-in…
Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
Hongxuan Zhang, Zhining Liu, Yao Zhao +4
In this work, we propose FastCoT, a model-agnostic framework based on parallel decoding without any further training of an auxiliary model or modification to the LLM itself. FastCo…