DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation
arXiv:2607.13967
The paper introduces DeltaMerge‑LowRes, a method that learns separate language and task parameter deltas and merges them in weight space using rules such as cross‑axis TIES, enabling low‑resource adaptation of multilingual encoders with only a few hundred examples.
Abstract
Adapting a multilingual encoder to a new language \emph{and} a new task with only a few hundred gold examples is a common low-resource NLP setting, yet the two axes are usually fused via an expensive language--task fine-tuning run. We ask whether they can instead be trained separately and recombined in weight space. \DeltaMergeLowRes{} learns a language delta from unlabeled monolingual text and a task delta from labeled English data, then composes them at inference under one of four rules: additive, activation-guided, sparsity-aware, and a novel \emph{cross-axis TIES}. The new rule adapts the TIES-Merging steps of trimming, sign election, and merging to the language and task axes rather than to two task axes. Holding fixed across rules on four task families and four African languages ( evaluated cells, -sample paired bootstrap per cell), we find: (i) cross-axis TIES wins summarisation on languages by to chrF (chrF vs.\ task-only); (ii) it improves QA F1 by and EM by ; and (iii) sparsity-aware merging cuts classification ECE by at parity macro-F1. The composition rule materially changes what the merged model preserves, suppresses, and calibrates. We release all JSON traces and a claim ledger.