63 citations · 84 across the 14 of their papers we have counts for
4 papers · 1 filter
Fast Tensorization of Neural Networks via Slice-wise Feature Distillation
Safa Hamreras, Sukhbinder Singh, Román Orús
We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on…
Weight Clustering Reveals the Importance of Relative Rank in Large Language Models
Borja Aizpurua, Sukhbinder Singh, Román Orús
Large language models (LLMs) contain billions of parameters, but many exact weight values are not essential. We apply scalar -means clustering to pretrained models, replacing ea…
Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing
Roman Rausch, David Jansen, Sukhbinder Singh +1
Large Language Models (LLMs) are very demanding in terms of their computational resources. Low-rank decompositions of LLM weights, e.g. via Singular Value Decomposition (SVD), is a…
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
Safa Hamreras, Sukhbinder Singh, Román Orús
Tensorizing a neural network involves reshaping some or all of its dense weight matrices into higher-order tensors and approximating them using low-rank tensor network decompositio…