2 citations · 5 across the 18 of their papers we have counts for
3 papers · 1 filter
Predicting and Repairing Merge Collapse in Large Language Models
Jungseob Lee, Seungyoon Lee, Sugyeong Eo +3
Large language models fine-tuned from a shared base can be merged by averaging their task vectors, but some merges collapse far below the base model, and common merge operators giv…
Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
Jungseob Lee, Seungyoon Lee, Seongtae Hong +2
At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with conte…
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
Sugyeong Eo, Jungjun Lee, Chanjun Park +1
A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational co…