4 papers · 1 filter
Exploring Scaling Laws for Local SGD in Large Language Model Training
Qiaozhi He, Xiaomin Zhuang, Zhihua Wu
This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive e…
ChuXin: 1.6B Technical Report
Xiaomin Zhuang, Yufan Jiang, Qiaozhi He +1
In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weigh…
Efficient LLM Inference with Kcache
Qiaozhi He, Zhihua Wu
Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the…
Code Comparison Tuning for Code Large Language Models
Yufan Jiang, Qiaozhi He, Xiaomin Zhuang +1
We present Code Comparison Tuning (CCT), a simple and effective tuning method for code large language models (Code LLMs) to better handle subtle code errors. Specifically, we integ…