A fingerprint is a compact value derived from data so a system can compare or identify it later. In crawling, fingerprints are commonly made from URLs or page content to detect repeats and changes. The exact method depends on what “the same” should mean for the task.
Pick the input carefully
A URL fingerprint may use a normalized URL: lowercase the hostname, remove a fragment, and handle known tracking parameters. A content fingerprint may hash selected fields after normalization. These two fingerprints answer different questions—whether the crawler has seen an address, and whether the content appears unchanged.
Cryptographic hashes are useful for exact matching but do not recognize near-duplicates. Small input changes produce different values. For similarity, use a technique designed for approximate comparison and be prepared for false matches.
Store the normalization rule or version with the fingerprint. Otherwise a later change to URL cleanup or text extraction can make old and new values look inconsistent. Do not treat a fingerprint as a reversible encoding; it cannot recreate the original data.
Leave a Reply