23 citations · 39 across the 10 of their papers we have counts for
13 papers
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-AI, :, Anyi Xu +585
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation…
PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation
Yida Wang, Xiuhong Li, Jianping Ma +9
Interconnecting geographically dispersed clusters over wide-area Ethernet provides a scalable and cost-effective alternative to dedicated intra-datacenter heterogeneous clusters fo…
HiFloat4 Format for Language Model Inference
Yuanyong Luo, Jing Huang, Yu Cheng +19
This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata…
db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism
Siqi Chen, Ke Hong, Tianchen Zhao +4
Scaling Diffusion Transformer (DiT) inference via sequence parallelism is critical for reducing latency in visual generation, but is severely hampered by workload imbalance when ap…
Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design
Zixiao Huang, Wen Zeng, Tianyu Fu +10
LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit t…
TASP: Topology-aware Sequence Parallelism
Yida Wang, Ke Hong, Xiuhong Li +4
Long-context large language models (LLMs) face constraints due to the quadratic complexity of the self-attention mechanism. The mainstream sequence parallelism (SP) method, Ring At…
PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
Tianchen Zhao, Ke Hong, Xinhao Yang +8
In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolu…
semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
Ke Hong, Lufang Chen, Zhong Wang +9
Existing large language model (LLM) serving systems fall into two categories: 1) a unified system where prefill phase and decode phase are co-located on the same GPU, sharing the u…
Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering
Ke Hong, Xiuhong Li, Minxu Liu +9
Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GP…
MBQ: Modality-Balanced Quantization for Large Vision-Language Models
Shiyao Li, Yingchun Hu, Xuefei Ning +10
Vision-Language Models (VLMs) have enabled a variety of real-world applications. The large parameter size of VLMs brings large memory and computation overhead which poses significa…
A Survey on Efficient Inference for Large Language Models
Zixuan Zhou, Xuefei Ning, Ke Hong +12
Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirem…
FlashDecoding++: Faster Large Language Model Inference on GPUs
Ke Hong, Guohao Dai, Jiaming Xu +6
As the Large Language Model (LLM) becomes increasingly important in various domains. However, the following challenges still remain unsolved in accelerating LLM inference: (1) Sync…
TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs
Haotian Tang, Shang Yang, Zhijian Liu +6
Sparse convolution plays a pivotal role in emerging workloads, including point cloud processing in AR/VR, autonomous driving, and graph understanding in recommendation systems. Sin…