Principle:ChenghaoMou Text dedup False Positive Verification SimHash

Knowledge Sources	text-dedup
Domains	Deduplication, Verification
Last Updated	2026-02-14 21:00 GMT

Overview

An optional post-clustering verification step that filters false positive duplicate pairs by computing exact Jaccard similarity within candidate clusters from SimHash.

Description

SimHash clustering based on Hamming distance can produce false positives where the binary fingerprints are close but the actual text overlap is below the desired threshold. This verification step: (1) groups all candidate duplicates by their assigned cluster, (2) computes exact Jaccard similarity for all pairs within each cluster, (3) re-clusters verified pairs using Union-Find, and (4) discards pairs below the jaccard_threshold.

Unlike the MinHash verification which uses Polars map_elements, the SimHash verification uses pure Python iteration over cluster groups with tqdm progress tracking.

Usage

Use this principle when SimHash deduplication requires high precision and the check_false_positive flag is enabled.

Theoretical Basis

Same as MinHash verification: exact Jaccard similarity computation. $J(A,B)=\frac{|A\cap B|}{|A\cup B|}$

Pairs are verified if J(A, B) >= jaccard_threshold. Verified pairs are re-clustered via Union-Find.

Related Pages

Implemented By

• Implementation:ChenghaoMou_Text_dedup_SimHash_Check_False_Positives

Uses Heuristic

• Heuristic:ChenghaoMou_Text_dedup_False_Positive_Verification_Tradeoff