Web search engine built from the CommonCrawl corpus

Show HN: Web search engine built from the CommonCrawl corpus

Web search engine built from the CommonCrawl corpus is a business in show hn launch.

Profile

Hello HN, I'm building an alternative web search engine. You can try the demo here: https://search.xorsoft.dev/My goal was to reach 1% of Google's index. This is 4 billion pages across 33.5 million domains. I was inspired by the earlier attempts shared here: "Building a web search engine from scratch with 3B neural embeddings" [1] and "Crawling a billion web pages in just over 24 hours" [2]. I decided not to replicate them and instead used CommonCrawl corpus, built a pipeline, and threw the results into a classic full-text search index.First, I got my hands on a dedicated Hetzner server with 4x10TB HDDs. It allowed me to crunch CommonCrawl data and stay within my hobby budget. The backend is

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

The company is listed at the Show HN launch stage of development.

Website: search.xorsoft.dev