4 papers · 1 filter
Sparse Weight Decomposition for Efficient Circuit Extraction
Chuanhao Yan, Xuhan Huang, Yawen Duan +4
Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations…
Hyperball May Not Be a Free Lunch
Yihao Xiao, Jialong Sun, Zitian Gao +5
For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing…
LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling
Jian Yang, Shawn Guo, Wei Zhang +16
Looped Transformers scale latent computation by repeatedly applying shared blocks, but sequential looping increases latency and KV-cache memory with the loop count. Parallel loop T…
Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization
Fei Bai, Zhipeng Chen, Chuan Hao +6
Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from ve…