9 papers
Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1
Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance requir…
Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivity
Kasun Dewage, Marianna Pensky, Suranadi De Silva
Many post-training quantization (PTQ) methods use layer-wise reconstruction, second-order proxy objectives, or activation-aware transformations to reduce quantization-induced error…
Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Kasun Dewage, Marianna Pensky, Suranadi De Silva +1
We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outli…
KRAFTY: Khatri-Rao Framework for Joint Cluster Recovery
Siyi Gao, Zachary Lubberts, Marianna Pensky
When multiple datasets describe complementary information about the same set of entities, for example, brain scans of an individual over time, global trade network across years, or…
LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
Kasun Dewage, Marianna Pensky, Suranadi De Silva +1
We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient…
Perfect Clustering in Very Sparse Diverse Multiplex Networks
Marianna Pensky
The paper studies the DIverse MultiPLEx Signed Generalized Random Dot Product Graph (DIMPLE-SGRDPG) network model (Pensky (2024)), where all layers of the network have the same col…
Scalable community detection in massive networks via predictive assignment
Subhankar Bhadra, Marianna Pensky, Srijan Sengupta
Massive network datasets are becoming increasingly common in scientific applications. Existing community detection methods encounter significant computational challenges for such m…
Davis-Kahan Theorem in the two-to-infinity norm and its application to perfect clustering
Marianna Pensky
Many statistical applications, such as the Principal Component Analysis, matrix completion, tensor regression and many others, rely on accurate estimation of leading eigenvectors o…
Signed Diverse Multiplex Networks: Clustering and Inference
Marianna Pensky
The paper introduces a Signed Generalized Random Dot Product Graph (SGRDPG) model, which is a variant of the Generalized Random Dot Product Graph (GRDPG), where, in addition, edges…