2 citations · 9 across the 12 of their papers we have counts for
11 papers · 1 filter
Model Merging in Pre-training of Large Language Models
Yunshui Li, Yiyuan Ma, Shen Yan +23
Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this pa…
Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
ByteDance Seed, :, Jiaze Chen +267
We introduce Seed1.5-Thinking, capable of reasoning through thinking before responding, resulting in improved performance on a wide range of benchmarks. Seed1.5-Thinking achieves 8…
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Liang Chen, Zekun Wang, Shuhuai Ren +24
Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning ta…
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
Run Luo, Haonan Zhang, Longze Chen +13
The development of Multimodal Large Language Models (MLLMs) has seen significant advancements with increasing demands in various fields (e.g., multimodal agents, embodied intellige…
Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs
Lei Zhang, Yunshui Li, Jiaming Li +7
Some recently developed code large language models (Code LLMs) have been pre-trained on repository-level code data (Repo-Code LLMs), enabling these models to recognize repository s…
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Minzheng Wang, Longze Chen, Cheng Fu +11
Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks fo…
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
Longze Chen, Ziqiang Liu, Wanwei He +3
Long-context modeling capabilities are important for large language models (LLMs) in various applications. However, directly training LLMs with long context windows is insufficient…
One-Shot Learning as Instruction Data Prospector for Large Language Models
Yunshui Li, Binyuan Hui, Xiaobo Xia +9
Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may comp…
Marathon: A Race Through the Realm of Long Context with Large Language Models
Lei Zhang, Yunshui Li, Ziqiang Liu +5
With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models' c…
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts
Yunshui Li, Binyuan Hui, ZhiChao Yin +3
Perceiving multi-modal information and fulfilling dialogues with humans is a long-term goal of artificial intelligence. Pre-training is commonly regarded as an effective approach f…
Self-Distillation with Meta Learning for Knowledge Graph Completion
Yunshui Li, Junhao Liu, Chengming Li +1
In this paper, we propose a selfdistillation framework with meta learning(MetaSD) for knowledge graph completion with dynamic pruning, which aims to learn compressed graph embeddin…