533 citations · 605 across the 21 of their papers we have counts for
4 papers · 1 filter
Generating Interpretable Networks using Hypernetworks
Isaac Liao, Ziming Liu, Max Tegmark
An essential goal in mechanistic interpretability to decode a network, i.e., to convert a neural network's raw weights to an interpretable algorithm. Given the difficulty of the de…
Growing Brains: Co-emergence of Anatomical and Functional Modularity in Recurrent Neural Networks
Ziming Liu, Mikail Khona, Ila R. Fiete +1
Recurrent neural networks (RNNs) trained on compositional tasks can exhibit functional modularity, in which neurons can be clustered by activity similarity and participation in sha…
Grokking as Compression: A Nonlinear Complexity Perspective
Ziming Liu, Ziqian Zhong, Max Tegmark
We attribute grokking, the phenomenon where generalization is much delayed after memorization, to compression. To do so, we define linear mapping number (LMN) to measure network co…
A Neural Scaling Law from Lottery Ticket Ensembling
Ziming Liu, Max Tegmark
Neural scaling laws (NSL) refer to the phenomenon where model performance improves with scale. Sharma & Kaplan analyzed NSL using approximation theory and predict that MSE losses d…