4 papers
Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle
Ruifeng Ren, Sheng Ouyang, Huayi Tang +1
Attention-based Transformers have demonstrated strong adaptability across a wide range of tasks and have become the backbone of modern Large Language Models (LLMs). However, their…
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity
Ruifeng Ren, Yong Liu
Compression has been a critical lens to understand the success of Transformers. In the past, we have typically taken the target distribution as a criterion to evaluate a model's co…
Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
Xinhao Yao, Ruifeng Ren, Yun Liao +2
Chain-of-Thought (CoT) training has markedly advanced the reasoning capabilities of large language models (LLMs), yet the mechanisms by which CoT training enhances generalization r…
Exploring the Limitations of Mamba in COPY and CoT Reasoning
Ruifeng Ren, Zhicong Li, Yong Liu
Transformers have become the backbone of modern Large Language Models (LLMs); however, their inference overhead grows linearly with the sequence length, posing challenges for model…