4 papers
Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models
Youngji Roh, Hyunjin Cho, Jaehyung Kim
Large Language Models (LLMs) exhibit highly anisotropic internal representations, often characterized by massive activations, a phenomenon where a small subset of feature dimension…
LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
Minju Gwak, Minseo Kwak, Dongseok Lee +3
Reinforcement learning (RL) post-training has shown to improve reasoning in large language models (LLMs). However, there has been little exploration on the problem of data contamin…
Revisiting the Uniform Information Density Hypothesis in LLM Reasoning
Minju Gwak, Guijin Son, Jaehyung Kim
The Uniform Information Density (UID) hypothesis proposes that effective communication is achieved by maintaining a stable flow of information. In this work, we revisit this princi…
Revisiting the UID Hypothesis in LLM Reasoning Traces
Minju Gwak, Guijin Son, Jaehyung Kim
Large language models (LLMs) often solve problems using step-by-step Chain-of-Thought (CoT) reasoning, yet these intermediate steps are frequently unfaithful or hard to interpret.…