1 citations · 1 across the 13 of their papers we have counts for
9 papers · 1 filter
EntropyLong: Effective Long-Context Training via Predictive Uncertainty
Junlong Jia, Ziyang Chen, Xing Wu +5
Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic…
LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
Junlong Jia, Xing Wu, Chaochen Gao +8
High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-b…
AdaMuon: Adaptive Muon Optimizer
Chongjie Si, Debing Zhang, Wei Shen
We propose AdaMuon, a novel optimizer that combines element-wise adaptivity with orthogonal updates for large-scale neural network training. AdaMuon incorporates two tightly couple…
dots.llm1 Technical Report
Bi Huo, Bin Tu, Cheng Qin +24
Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this…
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Chaochen Gao, Xing Wu, Zijia Lin +2
High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long…
Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge
Pengwei Tang, Xiaolin Hu, Yong Liu +4
Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs), but it still suffers from catastrophic forgetting. Recent work ha…
SedarEval: Automated Evaluation using Self-Adaptive Rubrics
Zhiyuan Fan, Weinong Wang, Xing Wu +1
The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (L…
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
Haotian Xu, Xing Wu, Weinong Wang +11
Can scaling transform reasoning? In this work, we explore the untapped potential of scaling Long Chain-of-Thought (Long-CoT) data to 1000k samples, pioneering the development of a…
NExtLong: Toward Effective Long-Context Training without Long Documents
Chaochen Gao, Xing Wu, Zijia Lin +2
Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synt…