4 papers
Exploring Scaling Laws for Local SGD in Large Language Model Training
Qiaozhi He, Xiaomin Zhuang, Zhihua Wu
This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive e…
ChuXin: 1.6B Technical Report
Xiaomin Zhuang, Yufan Jiang, Qiaozhi He +1
In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weigh…
Code Comparison Tuning for Code Large Language Models
Yufan Jiang, Qiaozhi He, Xiaomin Zhuang +1
We present Code Comparison Tuning (CCT), a simple and effective tuning method for code large language models (Code LLMs) to better handle subtle code errors. Specifically, we integ…
RecycleGPT: An Autoregressive Language Model with Recyclable Module
Yufan Jiang, Qiaozhi He, Xiaomin Zhuang +4
Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by…