5 papers
Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights
Irene Tallini, Daniele Solombrino, Alberto Cazzaniga +1
We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a st…
The Undetected Damage of Quantization on Retrieval and How to Fix It
Luca Zhou, Alessandro Zirilli, Daniele Solombrino +2
We show that a quantized model that keeps its classification accuracy still changes to of its top-1 retrieval results, and that aggregate ranking metrics reveal only pa…
Zero-Shot Quantization via Weight-Space Arithmetic
Daniele Solombrino, Antonio Andrea Gargiulo, Alessandro Zirilli +3
We show that robustness to post-training quantization (PTQ) is a transferable direction in weight space. We call this direction the quantization vector: extracted from a donor task…
On Task Vectors and Gradients
Luca Zhou, Daniele Solombrino, Donato Crisostomi +4
Task arithmetic has emerged as a simple yet powerful technique for model merging, enabling the combination of multiple finetuned models into one. Despite its empirical success, a c…
ATM: Improving Model Merging by Alternating Tuning and Merging
Luca Zhou, Daniele Solombrino, Donato Crisostomi +3
Model merging has emerged as a cost-efficient approximation to multitask learning. Among merging strategies, task arithmetic is notable for its simplicity and effectiveness. In thi…