4 citations · 4 across the 3 of their papers we have counts for
4 papers
Does self-distillation improve reasoning?
Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5
Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental fe…
Data Engineering for Scaling Language Models to 128K Context
Yao Fu, Rameswar Panda, Xinyao Niu +4
We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in part…
Diversity Measurement and Subset Selection for Instruction Tuning Datasets
Peiqi Wang, Yikang Shen, Zhen Guo +4
We aim to select data subsets for the fine-tuning of large language models to more effectively follow instructions. Prior work has emphasized the importance of diversity in dataset…
LangNav: Language as a Perceptual Representation for Navigation
Bowen Pan, Rameswar Panda, SouYoung Jin +4
We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision sy…