Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning

Show HN: Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning

Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning is a business in show hn launch.

Profile

I've been working on applying LLMs to long-context, verifiable problems over the past year, and today I'm releasing a benchmark of 62,000 pencil puzzles across 94 types (sudoku, nonori, slitherlink, etc.). The benchmark also allows for intermediate checks /rule breaks for all varieties at any step.I tested 51 models against a subset (300 puzzles) in two modes: single-shot (output the full solution) and agentic (iterate with verifier feedback).Some results:- Best model (GPT 5.2@xhigh) solves 56%. (~ half the puzzles are unsolved by any model)- Agentic solves average 29 turns. The longest attempt took ~1,200 turns over 14 hours.- Cost per success varies wildly (cheapest: $0.00033 — Grok 4.1 Fa

As a business in show hn launch, Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning is listed here with the public details available about it.

Directory data places it at the Show HN launch stage.

Website: ppbench.com