SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication

Embedding-based training data deduplication retrieves candidate duplicate edges above an application similarity threshold, but fixed-probe inverted-file (IVF) search ignores this predicate when giving every query the same partition budget. Across four Hunyuan workloads, qualifying neighbors appear e…