40 citations · 87 across the 27 of their papers we have counts for
7 papers · 1 filter
SkillRater: Untangling Capabilities in Multimodal Data
Naveen Sahi, Jeremy Dohmann, Armen Aghajanyan +1
Data curation methods typically assign samples a single quality score. We argue this scalar framing is fundamentally limited: when training requires multiple distinct capabilities,…
Improving MoE Compute Efficiency by Composing Weight and Data Sparsity
Maciej Kilian, Oleg Mkrtchyan, Luke Zettlemoyer +2
Mixture-of-Experts layers achieve compute efficiency through weight sparsity: each token activates only a subset of experts. Data sparsity, where each expert processes only a subse…
CoSMoEs: Compact Sparse Mixture of Experts
Patrick Huber, Akshat Shrivastava, Ernie Chang +3
Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mi…
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs
Charlie Hou, Akshat Shrivastava, Hongyuan Zhan +5
On-device training is currently the most common approach for training machine learning (ML) models on private, distributed user data. Despite this, on-device training has several d…
Privately Customizing Prefinetuning to Better Match User Data in Federated Learning
Charlie Hou, Hongyuan Zhan, Akshat Shrivastava +4
In Federated Learning (FL), accessing private client data incurs communication and privacy costs. As a result, FL deployments commonly prefinetune pretrained foundation models on a…
Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization
David Eriksson, Pierce I-Jen Chuang, Samuel Daulton +7
When tuning the architecture and hyperparameters of large machine learning models for on-device deployment, it is desirable to understand the optimal trade-offs between on-device l…
Better Fine-Tuning by Reducing Representational Collapse
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta +3
Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on t…