An estimation benchmark for language models

Estimation problems are interesting because they test capabilities that matter in the real world: recursive nature of subproblems, creativity in solutions, need for reasonable abstractions, broad knowledge, telling apart known from unknown, commonsense, etc.

Estimating a quantity you've never seen is recursive problem solving under uncertainty. A hard number splits into smaller ones, and those split again, until you reach quantities you can actually anchor. There's no fixed procedure for it, so reaching an answer means finding an angle: which quantity to lean on, which route to take across.

For a model to be proficient at this task, it will need to be fluent in a variety of operations — recall, decomposition, triangulation — and to know when to use which.

The benchmark is designed to test these capabilities, and to provide a clear signal of which models are better at them.