2 citations · 5 across the 7 of their papers we have counts for
7 papers
Learning from History: A Retrieval-Augmented Framework for Spatiotemporal Prediction
Hao Jia, Penghao Zhao, Hao Wu +3
Accurate and long-term spatiotemporal prediction for complex physical systems remains a fundamental challenge in scientific computing. While deep learning models, as powerful param…
TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation
Lin Sun, Guangxiang Zhao, Xiaoqi Jian +18
The challenge of reducing the size of Large Language Models (LLMs) while maintaining their performance has gained significant attention. However, existing methods, such as model di…
HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
Ran Yan, Youhe Jiang, Xiaonan Nie +3
Training large language models (LLMs) is a computationally intensive task, which is typically conducted in data centers with homogeneous high-performance GPUs. In this paper, we ex…
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Hailin Zhang, Xiaodong Ji, Yilin Chen +5
As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations o…
MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
Pinxue Zhao, Hailin Zhang, Fangcheng Fu +9
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges…
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
Shuaipeng Li, Penghao Zhao, Hailin Zhang +10
In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optim…
Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge
Bin Xiao, Chunan Shi, Xiaonan Nie +5
Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically,…