DimPO: Dimensionality Reduction for Attention using Preference Optimization

Vojtěch Lanz, Yufei Cui, Prasanna Parthasarathi

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model be…