5 papers · 1 filter
Structured Agent Distillation for Large Language Model
Jun Liu, Zhenglun Kong, Peiyan Dong +10
Large language models (LLMs) exhibit strong capabilities as decision-making agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical de…
Advancing time series completion via RFAMoE and MDFF
Ci Zhang, Huayu Li, Changdi Yang +6
Recent studies show that using diffusion models for time series signal reconstruction holds great promise. However, such approaches remain largely unexplored in the domain of medic…
Squat: Quant Small Language Models on the Edge
Xuan Shen, Peiyan Dong, Zhenglun Kong +9
A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters. This trend is driven by the increasing concerns over cloud costs,…
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
Jun Liu, Zhenglun Kong, Peiyan Dong +10
Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-t…
Rethinking Token Reduction for State Space Models
Zheng Zhan, Yushu Wu, Zhenglun Kong +6
Recent advancements in State Space Models (SSMs) have attracted significant interest, particularly in models optimized for parallel training and handling long-range dependencies. A…