11 papers
Why Heavy-Tailed Weight Spectra Predict Model Quality
Joseph Wilson, Chris van der Heide, Liam Hodgkinson +3
A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model qual…
Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics
Conrad Ainslie, Pedram Hassanzadeh, Michael W. Mahoney +1
Neural autoregressive models have rapidly emerged as powerful emulators of high-dimensional chaotic systems, yet their long-term instability and error growth remain poorly understo…
Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data
Collin Cranston, Zhichao Wang, Todd Kemp +1
Recent work in random matrix theory (RMT) has developed the notion of deterministic equivalents: typically linear surrogate models that approximate the spectral behavior of large n…
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4
Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in me…
Free Decompression with Algebraic Spectral Curves
Siavash Ameli, Chris van der Heide, Liam Hodgkinson +1
Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and…
Uncertainty-Aware Diagnostics for Physics-Informed Machine Learning
Mara Daniels, Liam Hodgkinson, Michael Mahoney
Physics-informed machine learning (PIML) integrates prior physical information, often in the form of differential equation constraints, into the process of fitting machine learning…
Spectral Estimation with Free Decompression
Siavash Ameli, Chris van der Heide, Liam Hodgkinson +1
Computing eigenvalues of very large matrices is a critical task in many machine learning applications, including the evaluation of log-determinants, the trace of matrix functions,…
Models of Heavy-Tailed Mechanistic Universality
Liam Hodgkinson, Zhichao Wang, Michael W. Mahoney
Recent theoretical and empirical successes in deep learning, including the celebrated neural scaling laws, are punctuated by the observation that many objects of interest tend to e…
Determinant Estimation under Memory Constraints and Neural Scaling Laws
Siavash Ameli, Chris van der Heide, Liam Hodgkinson +2
Calculating or accurately estimating log-determinants of large positive definite matrices is of fundamental importance in many machine learning tasks. While its cubic computational…
A Statistical Framework for Ranking LLM-Based Chatbots
Siavash Ameli, Siyuan Zhuang, Ion Stoica +1
Large language models (LLMs) have transformed natural language processing, with frameworks like Chatbot Arena providing pioneering platforms for evaluating these models. By facilit…
How many classifiers do we need?
Hyunsuk Kim, Liam Hodgkinson, Ryan Theisen +1
As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of mult…