5 papers
SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learning
Tao Liu, Tao Feng, Xiangheng Li +9
Recent Text-to-SQL systems increasingly rely on multi-turn interaction, execution feedback, and reinforcement learning. However, most existing methods use execution correctness onl…
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
Siquan Li, Kaiqi Jiang, Jiacheng Sun +1
Despite the prevalence of the attention sink phenomenon in Large Language Models (LLMs), where initial tokens disproportionately monopolize attention scores, its structural origins…
PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
Haonan Wang, Brian Chen, Siquan Li +4
Parameter-Efficient Fine-Tuning (PEFT) methods have become crucial for rapidly adapting large language models (LLMs) to downstream tasks. Prefix-Tuning, an early and effective PEFT…
SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
Yao Tong, Haonan Wang, Siquan Li +2
Fingerprinting Large Language Models (LLMs)is essential for provenance verification and model attribution. Existing fingerprinting methods are primarily evaluated after fine-tuning…
Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences
Siquan Li, Yao Tong, Haonan Wang +1
Transformers underpin modern large language models (LLMs) and are commonly assumed to be behaviorally unstructured at random initialization, with all meaningful preferences emergin…