1 citations · 1 across the 9 of their papers we have counts for
3 papers · 1 filter
MobileLLM-Pro Technical Report
Patrick Huber, Ernie Chang, Wei Wen +16
Efficient on-device language models around 1 billion parameters are essential for powering low-latency AI applications on mobile and wearable devices. However, achieving strong per…
AutoMixer: Checkpoint Artifacts as Automatic Data Mixers
Ernie Chang, Yang Li, Patrick Huber +4
In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these…
CoSMoEs: Compact Sparse Mixture of Experts
Patrick Huber, Akshat Shrivastava, Ernie Chang +3
Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mi…