5 papers
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Haebin Shin, Lei Ji, Xiao Liu +4
As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge. To address this, we propose Dynamix…
EpiCoder: Encompassing Diversity and Complexity in Code Generation
Yaoxiang Wang, Haoling Li, Xin Zhang +10
Existing methods for code generation use code snippets as seed data, restricting the complexity and diversity of the synthesized data. In this paper, we introduce a novel feature t…
Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
Haoling Li, Xin Zhang, Xiao Liu +4
Large language models (LLMs) have revolutionized lots of fields of research. Although it is well-known that fine-tuning is essential for enhancing the capabilities of LLMs, existin…
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
Zhenghao Lin, Zihao Tang, Xiao Liu +31
We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our metic…
Integrative Decoding: Improve Factuality via Implicit Self-consistency
Yi Cheng, Xiao Liang, Yeyun Gong +11
Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective…