From the 1 of 6 linked papers with an AI index.
6 papers
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28
The paper describes a production-ready inference system for the MiMo-V2.5 large language model family that combines hybrid sliding window attention, sparse mixture-of-experts, and…
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Wenhao Li, Jinhao Dong, Hailin Zhang +3
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budge…
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
Xiaodong Ji, Hailin Zhang, Fangcheng Fu +1
Many advanced Large Language Model (LLM) applications require long-context processing, but the self-attention module becomes a bottleneck during the prefilling stage of inference d…
Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
Haoyang Li, Fangcheng Fu, Hao Ge +7
As the scale of models and training data continues to grow, there is an expanding reliance on more GPUs to train large-scale models, which inevitably increases the likelihood of en…
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Hailin Zhang, Xiaodong Ji, Yilin Chen +5
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations o…
MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
Pinxue Zhao, Hailin Zhang, Fangcheng Fu +9
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges…