1 citations · 1 across the 9 of their papers we have counts for
9 papers
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Hanxian Huang, Igor Fedorov, Andrey Gromov +14
Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce nea…
Short Data, Long Context: Distilling Positional Knowledge in Transformers
Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4
Extending the context window of language models typically requires expensive long-context pre-training, posing significant challenges for both training efficiency and data collecti…
MobileLLM-Pro Technical Report
Patrick Huber, Ernie Chang, Wei Wen +16
Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong per…
AutoMixer: Checkpoint Artifacts as Automatic Data Mixers
Ernie Chang, Yang Li, Patrick Huber +4
In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these…
CoSMoEs: Compact Sparse Mixture of Experts
Patrick Huber, Akshat Shrivastava, Ernie Chang +3
Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mi…
Scaling Parameter-Constrained Language Models with Quality Data
Ernie Chang, Matteo Paltenghi, Yang Li +7
Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting…
CoDi: Conversational Distillation for Grounded Question Answering
Patrick Huber, Arash Einolghozati, Rylan Conway +6
Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in t…
Small But Funny: A Feedback-Driven Approach to Humor Distillation
Sahithya Ravi, Patrick Huber, Akshat Shrivastava +4
The emergence of Large Language Models (LLMs) has brought to light promising language generation capabilities, particularly in performing tasks like complex reasoning and creative…
Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
Zekun Li, Zhiyu Zoey Chen, Mike Ross +7
Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, thei…