Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing

Roman Rausch, David Jansen, Sukhbinder Singh +1

Large Language Models (LLMs) are very demanding in terms of their computational resources. Low-rank decompositions of LLM weights, e.g. via Singular Value Decomposition (SVD), is a…