DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights

Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris +2

We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. We propose an alternative way of structuring LLMs with weight sharing betwe…