4 papers · 1 filter
Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling
Changze Lv, Zhenghua Wang, Yiran Ding +9
Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresent…
Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
Shulin Huang, Yiran Ding, Junshu Pan +1
Enhancing the complex reasoning capabilities of Large Language Models (LLMs) attracts widespread attention. While reinforcement learning (RL) has shown superior performance for imp…
Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
Zhenghua Wang, Yiran Ding, Changze Lv +5
Although large language models (LLMs) have achieved significant progress in handling long-context inputs, they still suffer from the ``lost-in-the-middle'' problem, where crucial i…
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang +5
Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by n…