2 papers
cs.CL2025
OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Seungjun Shin, Jaehoon Oh, Dokwan Oh
Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each toke…
cs.LG2025
House of Cards: Massive Weights in LLMs
Jaehoon Oh, Seungjun Shin, Dokwan Oh
Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the…