8 papers
Steering Instruction Hierarchies at Inference Time
Siqi Zeng, Sewoong Lee, Han Zhao +1
Instruction hierarchies are a core safety assumption of language model deployment: higher priority inputs, such as system prompts, should override conflicting lower priority inputs…
ELF: Embedded Language Flows
Keya Hu, Linlu Qiu, Yiyang Lu +5
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing…
AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
Di Liu, Ruitian Wang, Chen Chen +6
As large language models scale to longer contexts, loading the growing KV cache during attention computation becomes a critical bottleneck. Previous work has shown that attention c…
When LRP Diverges from Leave-One-Out in Transformers
Weiqiu You, Siqi Zeng, Yao-Hung Hubert Tsai +2
Leave-One-Out (LOO) provides an intuitive measure of feature importance but is computationally prohibitive. While Layer-Wise Relevance Propagation (LRP) offers a potentially effici…
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
Siqi Zeng, Yifei He, Meitong Liu +5
Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse se…
Learning Structured Representations with Hyperbolic Embeddings
Aditya Sinha, Siqi Zeng, Makoto Yamada +1
Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most…
Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transport
Siqi Zeng, Sixian Du, Makoto Yamada +1
To embed structured knowledge within labels into feature representations, prior work [Zeng et al., 2022] proposed to use the Cophenetic Correlation Coefficient (CPCC) as a regulari…
An Empirical Study of Self-supervised Learning with Wasserstein Distance
Makoto Yamada, Yuki Takezawa, Guillaume Houry +4
In this study, we delve into the problem of self-supervised learning (SSL) utilizing the 1-Wasserstein distance on a tree structure (a.k.a., Tree-Wasserstein distance (TWD)), where…