1 citations · 1 across the 1 of their papers we have counts for
7 papers · 1 filter
MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
Zhenpeng Su, Xing Wu, Xue Bai +5
Generative language models are usually pretrained on large text corpus via predicting the next token (i.e., sub-word/word/phrase) given the previous ones. Recent works have demonst…
EntropyLong: Effective Long-Context Training via Predictive Uncertainty
Junlong Jia, Ziyang Chen, Xing Wu +5
Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic…
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Chaochen Gao, Xing Wu, Zijia Lin +2
High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long…
NExtLong: Toward Effective Long-Context Training without Long Documents
Chaochen Gao, Xing Wu, Zijia Lin +2
Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synt…
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
Chaochen Gao, Xing Wu, Qi Fu +1
Recent advancements in large language models (LLMs) have highlighted the importance of extending context lengths for handling complex tasks. While traditional methods for training…
HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus
Zhenpeng Su, Xing Wu, Wei Zhou +2
ChatGPT has garnered significant interest due to its impressive performance; however, there is growing concern about its potential risks, particularly in the detection of AI-genera…