5 papers · 1 filter
Why Heavy-Tailed Weight Spectra Predict Model Quality
Joseph Wilson, Chris van der Heide, Liam Hodgkinson +3
A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model qual…
Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics
Conrad Ainslie, Pedram Hassanzadeh, Michael W. Mahoney +1
Neural autoregressive models have rapidly emerged as powerful emulators of high-dimensional chaotic systems, yet their long-term instability and error growth remain poorly understo…
Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data
Collin Cranston, Zhichao Wang, Todd Kemp +1
Recent work in random matrix theory (RMT) has developed the notion of deterministic equivalents: typically linear surrogate models that approximate the spectral behavior of large n…
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4
Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in me…
Free Decompression with Algebraic Spectral Curves
Siavash Ameli, Chris van der Heide, Liam Hodgkinson +1
Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and…