Actual Claude Tokenizer

Show HN: Actual Claude Tokenizer

Operating in show hn launch, Actual Claude Tokenizer operates.

Profile

I've seen a few "Claude tokenizers" floating around lately with all the 4.7 chatter, but most of them just hit the count_tokens endpoint and hand you back a number. You don't actually see how your text gets split or understand the changes from 4.6 to 4.7.I built this a while back for doing some mech interp research. It faithfully represents Claude token splitting - showing hidden tokens, real boundaries and so on. It is not cheap to run - essentially n^2 cost - you could optimise for longer sequences but you are not guaranteed a faithful representation if so.Open Source: https://github.com/R0bk/claude-tokenizerFeedback welcome, let me know if there are any edge cases that look wrong.P.S. I'd

The company sits in the show hn launch segment, where it competes on its core product or service.

The company is listed at the Show HN launch stage of development.

Website: tokenizer.robkopel.me