57 citations · 253 across the 32 of their papers we have counts for
34 papers
Accelerating Cross-Model LLM Inference via Attention Map Reuse
Zhan Jin, Yuze Luo, Bin Cui +1
LLM-augmented data workflows often involve multiple heterogeneous models operating over substantial shared context, such as schemas, tool specifications, interaction history, and i…
DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decoding
Yaojie Zhang, Linfeng Zhang, Bin Cui +1
Block diffusion speculative decoding improves LLM inference efficiency by proposing a block of future tokens in parallel and verifying them with a single forward pass through the t…
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
Siwei Chen, Siqi Chen, Xupeng Miao +1
Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing me…
Adaptive Resource Management and Quality Control for Streaming Video Generation
Yifei Xia, Hao Yuan, Suhan Ling +5
Autoregressive diffusion transformers (AR-DiTs) recast video generation from an offline paradigm to a real-time streaming one: the model generates video one chunk at a time, making…
Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training
Yuming Zhou, Haoyang Li, Sheng Lin +6
Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge.…
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
Chunan Shi, Yilei Chen, Yilin Chen +2
Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation. While approximate KV cache retention techniques reduce memory usage b…
AutoSci: A Memory-Centric Agentic System for the Full Scientific Research Lifecycle
Weitong Qian, Beicheng Xu, Zhongao Xie +16
Scientific research has traditionally been human-intensive, requiring researchers to coordinate literature, ideas, experiments, manuscripts, and review responses across long projec…
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
Yujie Wang, Siwei Chen, Longzan Luo +4
Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. W…
TridentServe: A Stage-level Serving System for Diffusion Pipelines
Yifei Xia, Fangcheng Fu, Hao Yuan +6
Diffusion pipelines, renowned for their powerful visual generation capabilities, have seen widespread adoption in generative vision tasks (e.g., text-to-image/video). These pipelin…
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Haoyang Li, Fangcheng Fu, Hao Ge +6
The Single-Program Multiple-Data (SPMD) paradigm provides a unified abstraction to annotate various parallel dimensions in distributed deep learning (DL) training. With SPMD, users…
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao +6
Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However…
LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
Xiaonan Nie, Qibin Liu, Fangcheng Fu +6
Larger transformer models always perform better on various tasks but require more costs to scale up the model size. To efficiently enlarge models, the mixture-of-experts (MoE) arch…
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
Yujie Wang, Shenhan Zhu, Fangcheng Fu +6
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However…
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Hailin Zhang, Xiaodong Ji, Yilin Chen +5
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations o…
Generative Dense Retrieval: Memory Can Be a Burden
Peiwen Yuan, Xinglin Wang, Shaoxiong Feng +5
Generative Retrieval (GR), autoregressively decoding relevant document identifiers given a query, has been shown to perform well under the setting of small-scale corpora. By memori…
SpotServe: Serving Generative Large Language Models on Preemptible Instances
Xupeng Miao, Chunan Shi, Jiangfei Duan +4
The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for…
Experimental Analysis of Large-scale Learnable Vector Storage Compression
Hailin Zhang, Penghao Zhao, Xupeng Miao +4
Learnable embedding vector is one of the most important applications in machine learning, and is widely used in various database-related domains. However, the high dimensionality o…
Model-enhanced Vector Index
Hailin Zhang, Yujing Wang, Qi Chen +16
Embedding-based retrieval methods construct vector indices to search for document representations that are most similar to the query representations. They are widely used in docume…
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Yujie Wang, Youhe Jiang, Xupeng Miao +5
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-…
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Youhe Jiang, Fangcheng Fu, Xupeng Miao +2
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model re…
Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference
Zihao Yu, Haoyang Li, Fangcheng Fu +2
Due to the recent success of diffusion models, text-to-image generation is becoming increasingly popular and achieves a wide range of applications. Among them, text-to-image editin…
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Xiaonan Nie, Xupeng Miao, Zilong Wang +5
With the increasing data volume, there is a trend of using large-scale pre-trained models to store the knowledge into an enormous number of model parameters. The training of these…
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Xiaonan Nie, Yi Liu, Fangcheng Fu +5
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as…
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang +4
Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) mode…