Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Emanuel Ben Baruch, Sivan Biham, Alon Shoshan +2

When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model's own zero-shot performance. In this regime, we uncover a surpri…