DIVE: Embedding Compression via Self-Limiting Gradient Updates
arXiv:2605.20689
The paper introduces DIVE, a residual compression adapter that reduces the dimensionality of language‑model embeddings using a self‑limiting hinge loss, geometry distillation, and head‑wise NT‑Xent, achieving strong retrieval performance on BEIR benchmarks while keeping inference lightweight.
Abstract
High-dimensional language-model embeddings increase storage and search costs, while supervised compressors can overfit when relevance labels are scarce. We present DIVE (Dimensionality reduction with Implicit View Ensembles), a residual compression adapter codesigned with a self-limiting hinge loss, geometry distillation, and head-wise NT-Xent over implicit coordinate views. The hinge stops updating satisfied ranking constraints, while the dense objectives stabilize the compressed representation; only the first head is retained at inference. Under query-disjoint evaluation with two LLM2Vec backbones, five BEIR benchmarks, 128d and 256d outputs, and six baselines, DIVE is the strongest adapter on all five primary benchmarks. It also outperforms PCA and an autoencoder in comparisons against unsupervised compressors.