4 papers
A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields
Tiexin Ding
Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study th…
Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training
Tiexin Ding
A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape is stable across layers and models, so the scale …
Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics
Tiexin Ding
Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter grows, overshoots, and then relax…
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
Tiexin Ding
We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers.…