6 papers · 1 filter
Convex Dataset Valuation for Post-Training
Siqi Zeng, Christopher Jung, Rui Li +7
Improving LLM performance on downstream tasks sometimes requires leveraging auxiliary datasets during post-training. In practice, however, developers face constraints on compute, l…
When LRP Diverges from Leave-One-Out in Transformers
Weiqiu You, Siqi Zeng, Yao-Hung Hubert Tsai +2
Leave-One-Out (LOO) provides an intuitive measure of feature importance but is computationally prohibitive. While Layer-Wise Relevance Propagation (LRP) offers a potentially effici…
MergeBench: A Benchmark for Merging Domain-Specialized LLMs
Yifei He, Siqi Zeng, Yuzheng Hu +3
Model merging provides a scalable alternative to multi-task training by combining specialized finetuned models through parameter arithmetic, enabling efficient deployment without t…
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
Siqi Zeng, Yifei He, Meitong Liu +5
Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse se…
Learning Structured Representations with Hyperbolic Embeddings
Aditya Sinha, Siqi Zeng, Makoto Yamada +1
Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most…
Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transport
Siqi Zeng, Sixian Du, Makoto Yamada +1
To embed structured knowledge within labels into feature representations, prior work [Zeng et al., 2022] proposed to use the Cophenetic Correlation Coefficient (CPCC) as a regulari…