6 papers
Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1
Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance requir…
Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivity
Kasun Dewage, Marianna Pensky, Suranadi De Silva
Many post-training quantization (PTQ) methods use layer-wise reconstruction, second-order proxy objectives, or activation-aware transformations to reduce quantization-induced error…
Hybrid Neural-Classical Correction for Frozen Time Series Foundation Models: A Comprehensive Ablation Study on High-Frequency Stock Prediction
Kasun Dewage, Suranadi De Silva, Shankhadeep Mondal
Foundation models for time series forecasting demonstrate impressive zero-shot generalization but often underperform on specialized domains such as high-frequency finance. We prese…
Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Kasun Dewage, Marianna Pensky, Suranadi De Silva +1
We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outli…
LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
Kasun Dewage, Marianna Pensky, Suranadi De Silva +1
We introduce LoRA-CRAFT (\textbf{C}ross-layer \textbf{R}ank \textbf{A}daptation via \textbf{F}rozen \textbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient…
Refined upper bounds for the numerical radius via weighted operator means
Shankhadeep Mondal, Ram Narayan Mohapatra, Kasun Tharuka Dewage
We establish a parameterized family of upper bounds for the numerical radius of bounded linear operators on a complex Hilbert space, based on weighted expressions involving the mod…