4 papers · 1 filter
LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning
Md Kowsher, Haris Mansoor, Nusrat Jahan Prottasha +4
MoE-PEFT methods combine Mixture of Experts with parameter-efficient fine-tuning for multi-task adaptation, but require separate adapters per expert causing trainable parameters to…
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
Guanghui Min, Tianhao Huang, Ke Wan +1
Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific ta…
Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
Chen Chen, Lai Wei
Large language model (LLM) scaling is hitting a wall. Widening models yields diminishing returns, and extending context length does not improve fundamental expressivity. In contras…
Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
Nusrat Jahan Prottasha, Md Kowsher, Chun-Nam Yu +2
Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing me…