Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms

Ziwei Su, Junyu Ren, Victor Veitch

Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. Howev…