Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

Valentijn Oldenburg, Floris de Kam, Bente Zuijdam +4

Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper inve…