4 papers · 1 filter
Fast Tensorization of Neural Networks via Slice-wise Feature Distillation
Safa Hamreras, Sukhbinder Singh, Román Orús
We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on…
Only relative ranks matter in weight-clustered large language models
Borja Aizpurua, Sukhbinder Singh, Román Orús
Large language models (LLMs) contain billions of parameters, yet many exact values are not essential. We show that what matters most is the relative rank of weights-whether one con…
Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing
Roman Rausch, David Jansen, Sukhbinder Singh +1
Large Language Models (LLMs) are very demanding in terms of their computational resources. Low-rank decompositions of LLM weights, e.g. via Singular Value Decomposition (SVD), is a…
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
Safa Hamreras, Sukhbinder Singh, Román Orús
Tensorizing a neural network involves reshaping some or all of its dense weight matrices into higher-order tensors and approximating them using low-rank tensor network decompositio…