5 papers · 1 filter
The Optimal Decoder is Tilted Temperature
Nikolaos Papagiannis, Ananth Grama, Wojciech Szpankowski
Larger sample budgets benefit from higher sampling temperatures in LLMs, but what temperature tuning optimizes and why it works remain poorly understood. We seek to answer these qu…
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Linkai Ma, Xinyu Luo, Mengbo Wang +3
The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, motivated by an RMS-stability argum…
Certified Approximation for Interpretable Representer Landmarks
Jayanta Mukherjee, Shourya Verma, Mengbo Wang +2
Representer explanations rank the training landmarks that most influence a self-supervised representation. At scale, this ranking rests on up to four stacked approximations of the…
Inference-Time Code Selection via Symbolic Equivalence Partitioning
David Cho, Yifan Wang, Fanping Sui +1
Sampling multiple candidate programs at inference time is an effective way to improve LLM code generation. However, its benefit depends on reliably selecting a correct solution fro…
Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
Bolian Li, Yifan Wang, Yi Ding +3
Reinforcement learning (RL) has enabled complex reasoning abilities in large language models (LLMs). However, most RL algorithms suffer from performance saturation, preventing cont…