This page is the standing record of how Blender-Bench runs, scores, and reports. It is meant to be read alongside the leaderboard: every metric there is defined here, and the honest limits are stated plainly rather than buried.
Every contestant faces an identical scaffold. A single, versioned benchmark system prompt is composed with the harness' execution rules and, for some categories, a small category addendum — the same composition for every model on a task. A contestant is a model plus its reasoning variant; nothing about the task setup is tuned per model. Each model drives a real Blender entirely through the Blender MCP tool interface — no human hands on the mouse.
Contestants work through the Blender MCP tool surface: scene and object inspection, a viewport screenshot (offered only to vision-capable models — text-only models never receive it), and a Python execution tool that runs bpy against the live scene. Tool calls execute one at a time, in the order issued, against a single Blender instance — each call sees the state the previous one left, and issuing several at once buys no parallelism.
Each run has a budget of 50 tool calls and 30 minutes of wall-clock time by default. A task may override either in its frontmatter when the work genuinely needs more or less. A run that reaches its budget having produced a scene is still a completed run; it is not a failure.
Prompts and tasks are versioned, and every run records which versions it ran. The shared system prompt and category addenda carry version ids; the harness prompt carries its own (for example raw-v3 / oneshot-v3). Task prompt bodies are versioned too: editing a task's prompt archives the previous body immutably under tasks/_versions/, so a run always points at the exact wording it was given. Old runs keep their original provenance — nothing is back-dated.
When resources allow, a contestant runs a task more than once. Every successful run enters the arena — there is no cherry-picked showcase run and no discarded median. Runs that fail are preserved on disk and counted into the published completion rate; they are simply not offered up for pairwise voting.
These are two different signals and the leaderboard shows both. Done % is self-declared: the fraction of a contestant's runs where the model itself signalled it was finished. Completion rate is measured from the run's recorded status: the fraction that finished cleanly (status ok, which includes a run that used its full budget but still produced a scene) rather than ending in a model error, a crash, a hang, or an environment error. A model can declare itself done on a broken scene, or finish a good scene without ever declaring done — so the two numbers can diverge, and that gap is itself informative.
The interactive 3D view is the strongest "this is real 3D" evidence the site can offer, but it is a best-effort glTF export, not the authored .blend. glTF cannot represent every Blender feature — area, world, and emissive lights are dropped by the exporter, so a lighting sidecar restores them in the SCENE-LIGHTS view, and a separate baked preview export carries materials that the raw exporter cannot. Features glTF simply cannot encode are not expected to survive it; judge the final render as the authoritative image and treat the viewer as an inspection aid.
The final benchmark capture overrides whatever render settings the contestant left behind and renders every scene the same way: Cycles, 1536×1536, 64 samples, with denoising and GPU acceleration (CPU fallback), plus a short 4-frame turntable. Rendering everyone through one fixed pipeline keeps the comparison about the scene, not about who tuned their output settings.
Voting is blind: pairs are shown without names, asset URLs go through opaque tokens, and the page carries no run ids until you have voted. But blinding is not perfect — a model with a distinctive style can be recognized by a practiced eye, and we would rather disclose that style recognition leak than pretend it does not exist. We treat the aggregate of many voters as the signal, not any single pair.
Each run records the environment it ran in: the Blender version, the MCP addon and server versions, and the render device. When the toolchain moves, the record shows which runs predate the change instead of silently mixing eras.
Two things are designed but not yet live, and are called out here so the current numbers are read for what they are. A hidden / rotating task set — held back from the public library — will guard against tasks being optimized against directly. And per-task requirement checklists (core / supporting / bonus) will add a structured, non-vote scoring axis alongside the pairwise ratings. Until then, the public task set and human pairwise voting are the whole of the score.
Back to about · the leaderboard · the task library.