7 papers
Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models
Zeyu Gan, Ruifeng Ren, Wei Yao +9
The rapid emergence of Large Language Models (LLMs) has precipitated a profound paradigm shift in Artificial Intelligence, delivering monumental engineering successes that increasi…
Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge
Pengwei Tang, Xiaolin Hu, Yong Liu +4
Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs), but it still suffers from catastrophic forgetting. Recent work ha…
ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Pengwei Tang, Xiaolin Hu, Yong Liu
Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Models (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended t…
The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
Xinhao Yao, Lu Yu, Xiaolin Hu +4
The ongoing debate on whether reinforcement learning with verifiable rewards (RLVR) expands or shrinks the reasoning capabilities of large language models (LLMs) remains unresolved…
Chemical knowledge-informed framework for privacy-aware retrosynthesis learning
Guikun Chen, Xu Zhang, Xiaolin Hu +3
Chemical reaction data is a pivotal asset, driving advances in competitive fields such as pharmaceuticals, materials science, and industrial chemistry. Its proprietary nature rende…
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization
Xinhao Yao, Hongjin Qian, Xiaolin Hu +5
Large Language Models (LLMs), built on Transformer architectures, exhibit remarkable generalization across a wide range of tasks. However, fine-tuning these models for specific tas…