SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication

Zhisheng Hu, Zhifang Li, Junjie Chen +6

Embedding-based training data deduplication retrieves candidate duplicate edges above an application similarity threshold, but fixed-probe inverted-file (IVF) search ignores this p…