4 papers
Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
Yitao Jiang, Yaoqing Yang, Luyang Zhao +2
Long-context Transformer inference increasingly relies on KV-cache compression or quantization. Prior rotation and transform-coding results suggest that the channel basis of each k…
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3
We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions. In particular, we derive the exact closed-for…
Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
Yuxin Wang, Yuanzhe Hu, Xiaokun Zhong +7
Neural networks trained under different hyperparameter settings can fall into distinct training "regimes," with consistent behavior within regimes and qualitative differences acros…
Spectral Insights into Data-Oblivious Critical Layers in Large Language Models
Xuyuan Liu, Lei Hsiung, Yaoqing Yang +1
Understanding how feature representations evolve across layers in large language models (LLMs) is key to improving their interpretability and robustness. While recent studies have…