Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken

Show HN: Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken

Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken is a company in show hn launch.

Profile

Hey HN,We're ZSpaceLabs, Burn framework contributors working to make Rust a first-class AI/ML stack.We just released wordchipper, a Rust-native BPE tokenizer for the OpenAI GPT-2 tokenizer family (r50k, cl100k, o200k). On a 64-core machine with the o200k vocab (GPT-4o / GPT-5 tokenizer), we measured 2.4 GiB/s, about 9.2× faster than tiktoken-rs. Through the Python bindings it is typically 2–4× faster than tiktoken, depending on thread count.The main design goal was to make the internals easy to swap. The tokenizer is split into two parts: pre-tokenization (lexer) and BPE span encoding. Each part can be replaced independently, which makes it easy to experiment with different combinations of l

The company sits in the show hn launch segment, where it competes on its core product or service.

The company is listed at the Show HN launch stage of development.

Website: zspacelabs.ai/wordchipper/articles/substitutable/GitHub: