Fastdedup – Rust dataset deduplication (2:55 vs. 7:55 688MB vs. 22GB)

Show HN: Fastdedup – Rust dataset deduplication (2:55 vs. 7:55 688MB vs. 22GB)

Fastdedup – Rust dataset deduplication (2:55 vs. 7:55 688MB vs. 22GB) is a business in show hn launch.

Profile

I've been working on a Rust CLI for dataset deduplication and wanted to share benchmark results. Ran on FineWeb sample-10BT (14.8M records, 29GB) on a single machine. Exact dedup vs DuckDB + SHA-256:2:55 vs 7:55 wall clock (2.7x faster) 688MB vs 21.9GB peak RAM (32x less) Single core vs 4+ cores Duplicate counts match exactly (51,392 both ways)Fuzzy dedup (MinHash + LSH) vs datatrove:36:44 vs 3h50m+ — datatrove stage 1 alone ran for 3h50m and we killed it datatrove's bottleneck turned out to be spaCy word tokenization on every document before shingling. fastdedup uses character n-grams directly which is significantly cheaper 23GB vs 1.1GB RAM — this is a real trade-off, not a win. datatrove

The company sits in the show hn launch segment, where it competes on its core product or service.

Its current recorded stage is Show HN launch.

Website: wapplewhite4.github.io/fastdedup