FretBench – I tested 14 LLMs on reading guitar tabs. Most failed

Show HN: FretBench – I tested 14 LLMs on reading guitar tabs. Most failed

FretBench – I tested 14 LLMs on reading guitar tabs. Most failed is a business in show hn launch.

Profile

I built FretBench after noticing Gemini was confidently wrong about basic guitar tab questions. Tab is arguably the simplest notation in music: six lines, numbers for frets, read left to right. So I made a benchmark out of it.182 test cases, 4 tunings, 14 models via OpenRouter. Two open-weight Qwen models from Alibaba crushed everything else (83.5%), while most "flagship" models scored below 50%. MiniMax M2.5 scored worse than random guessing.Everything is open source: https://github.com/jmcapra/FretBenchI'm curious whether the performance gap is related to tokenisation of ASCII art — if anyone has insights on how different tokenisers handle grid-structured text, I'd love to hear it.

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

Directory data places it at the Show HN launch stage.

Website: fretbench.tymo.ai/blog/i-asked-14-ai-models-to-read-guitar-tabs