Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Chenyu Zhu, Ruoyu Zhao, Zhichao Lu

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute…