127 citations · 166 across the 10 of their papers we have counts for
Showing 2024Show all
3 papers · 1 filter
cs.CL2024
Base of RoPE Bounds Context Length
Xin Men, Mingyu Xu, Bingning Wang +4
Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation…
cs.CL2024★ 2 cited
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Zican Dong, Junyi Li, Xin Men +5
Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of t…
cs.CL2024★ 15 cited
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Xin Men, Mingyu Xu, Qingyu Zhang +5
As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. H…