Infini-News – 1.36B news articles from Common Crawl, queryable in ms

Show HN: Infini-News – 1.36B news articles from Common Crawl, queryable in ms

Operating in show hn launch, Infini-News – 1.36B news articles from Common Crawl, queryable in ms operates.

Profile

Infini-News is ten years of CC-NEWS (the news subset of Common Crawl), cleaned, enriched and turned into a full-text index so you can count any keyword or phrase across 1.36B articles in sub-second time (ok, now maybe a few seconds, but circumstantial), without downloading anything. It's free and open on Hugging Face. I did it because I was sick of having to manually scrape news websites and the like for research purposes and because it felt interesting personally to tackle a project of this scale. On top of data cleaning, we have run language, country (via TLDs and some other heuristics) and topic tagging over all the articles and I have indexed all of them using a recent new n-gram indexin

The company sits in the show hn launch segment, where it competes on its core product or service.

Its current recorded stage is Show HN launch.

Website: cs2.uni-graz.at/blog/infini-news