5 papers · 1 filter
Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
Bowei He, Lihao Yin, Huiling Zhen +5
Post-training compression has been a widely employed approach to scale down large language model (LLM) and facilitate efficient inference. In various proposed compression methods,…
Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging
Han Wu, Yuxuan Yao, Shuqi Liu +7
The transition from System 1 to System 2 reasoning in large language models (LLMs) has marked significant advancements in handling complex tasks through deliberate, iterative think…
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models
Shuqi Liu, Han Wu, Bowei He +3
Recent advances in large language models have led to numerous task-specialized fine-tuned variants, creating a need for efficient model merging techniques that preserve specialized…
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging
Zehua Liu, Han Wu, Yuxuan Yao +4
While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its abil…
1bit-Merging: Dynamic Quantized Merging for Large Language Models
Shuqi Liu, Yuxuan Yao, Bowei He +5
Recent advances in large language models have led to specialized models excelling in specific domains, creating a need for efficient model merging techniques. While traditional mer…