From the 1 of 4 linked papers with an AI index.
3 papers · 1 filter
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Mohammed Sabry, Sean Augenstein, Keith Rush +1
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixt…
SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei +4
SkillSmith is a method that lets a large language model reason over both textual knowledge and prefix‑tuned model weights, enabling it to generate new parametric skill prefixes for…
Context Training with Active Information Seeking
Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng +4
Most existing large language models (LLMs) are expensive to adapt after deployment, especially when a task requires newly produced information or niche domain knowledge. Recent wor…