14 papers · 1 filter
DREAM: Dense Retrieval Embeddings via Autoregressive Modeling
Yixuan Tang, Yi Yang
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labe…
Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
Yixuan Tang, Yi Yang
Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update $Î…
Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
Zecheng Tang, Quantong Qiu, Yi Yang +6
The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attent…
KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs
Yixuan Tang, Yi Yang
While LLMs are powerful embedding backbones, their application in training-free settings faces two structural challenges: causal attention restricts early tokens from accessing sub…
SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
Yixuan Tang, Yi Yang
Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective…
GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings
Yixuan Tang, Yi Yang
Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often a…