1 citations · 1 across the 9 of their papers we have counts for
6 papers · 1 filter
Short Data, Long Context: Distilling Positional Knowledge in Transformers
Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4
Extending the context window of language models typically requires expensive long-context pre-training, posing significant challenges for both training efficiency and data collecti…
AutoMixer: Checkpoint Artifacts as Automatic Data Mixers
Ernie Chang, Yang Li, Patrick Huber +4
In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these…
Scaling Parameter-Constrained Language Models with Quality Data
Ernie Chang, Matteo Paltenghi, Yang Li +7
Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting…
CoDi: Conversational Distillation for Grounded Question Answering
Patrick Huber, Arash Einolghozati, Rylan Conway +6
Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in t…
Small But Funny: A Feedback-Driven Approach to Humor Distillation
Sahithya Ravi, Patrick Huber, Akshat Shrivastava +4
The emergence of Large Language Models (LLMs) has brought to light promising language generation capabilities, particularly in performing tasks like complex reasoning and creative…
Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
Zekun Li, Zhiyu Zoey Chen, Mike Ross +7
Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, thei…