MineBench

A benchmark for evaluating AI spatial reasoning through Minecraft-style voxel construction.

Models are given a natural-language prompt and must produce raw 3D coordinates as JSON. In tool mode, models call voxel.exec (minimal primitives: block, box, line) to generate large builds beyond token-only JSON limits. MineBench visualizes the output and ranks models from blind head-to-head votes using a global Bradley-Terry model with uncertainty intervals.

Try it live

Note

MineBench is not technically a 'benchmark' as it has no objectively correct answers; it is a take on the LMSYS Chatbot Arena. Many use MineBench to get the general feel or "vibe" of a model. AI labs may use MineBench to privately A/B test model checkpoints.

Why MineBench?

Most LLM benchmarks test text and raw accuracy. MineBench instead tests whether a model can reason about 3D space. Given a prompt like "a medieval castle with four towers", the model must mentally construct geometry, pick materials, and output thousands of precise block coordinates. No vision model or diffusion – just math and spatial logic.

As it turns out, this kind of spatial reasoning correlates strongly with a model's raw general intelligence; the MineBench leaderboard tracks, anecdotally, the same hierarchy that most people observe in real-world usage: the smartest reasoning models are clearly visible when asked to produce visual builds.

MineBench, unlike other benchmarks, gives an easy way to visually determine (at least one aspect of) a model's raw intelligence. The ranking system also highlights which models are clearly 'bench-maxed' (i.e. when a model has amazing benchmarks on paper, but clearly lacks in real world usage).

Features

• Arena — blind head-to-head comparisons of pre-generated builds with confidence-aware ranking
• Sandbox — compare existing builds, generate new ones, or import output from any model
• Gallery — explore community prompts and keep signed-in generations
• Leaderboard — live rankings with win/loss/draw stats across all models
• Exports — save builds as GLB, STL, or WorldEdit .schem for Blender, 3D printing, and Minecraft

Documentation

• Full docs index: docs/README.md
• Local development: docs/local-development.md
• Operations and API reference: docs/operations.md
• Gallery and saved generations: docs/gallery.md
• Arena ranking: docs/arena-ranking-system.md
• Build export and imports: docs/build-export-import.md

Frequently Asked Questions

The full FAQ is available at minebench.ai/faq. Every answer has a stable link for sharing or citation.

About MineBench

• What is MineBench?
• How do models actually create the builds?
• Why do some models add objects or scenery that were not explicitly requested?

Methodology

• How are models ranked if there is no single correct build?
• Can models train on MineBench or “benchmax” it?
• How do grid size, block limits, and different leaderboard settings work?
• Why not add more prompts, grid sizes, block-limited settings, and other evaluation modes?
• Are generations one-shot?

Using MineBench

• Can I compare different models directly?
• Can models that are not on the official leaderboard be tested?
• Why isn't a particular model on the leaderboard?
• Can MineBench builds be exported?
• Is MineBench using Minecraft MCP, Blender MCP, or a coding agent?
• How can MineBench be supported or contributed to?

Supported Models

MineBench currently benchmarks models from OpenAI, Anthropic, Google, Moonshot, DeepSeek, MiniMax, xAI, Z.AI, Qwen, Meta, and any model available through OpenRouter.

Quick Start (Local)

This path lets you run the full app and compare existing builds from uploads/ without generating new ones.

Prereqs: Node.js 18+, pnpm, Docker.

pnpm install
cp .env.example .env
pnpm dev:setup

In a second terminal:

pnpm prompt --import

Then open:

• http://localhost:3000/ (Arena)
• http://localhost:3000/sandbox
• http://localhost:3000/leaderboard

For environment variables, live generation, seeding/import workflows, batch generation, API routes, troubleshooting, and deployment, see the docs:

• docs/local-development.md
• docs/operations.md
• docs/deployment.md

Sponsors

A huge thank you to the sponsors helping make MineBench possible:

• 3D-Agent
  • AI-powered tools for Blender and 3D workflows
  • 10% off with code MINEBENCH10
• OpenAI
• Anthropic
• Google DeepMind
• Z.ai
• Moonshot AI

Their support, including API credits, helps fund MineBench evaluations. If you would like to support MineBench yourself, you can support us here.

Contributing

Contributions are welcome! See CONTRIBUTING.md for how to add new models, submit benchmark prompts, improve the UI, or fix bugs.

Licenses

MIT

Texture pack: Faithful (see assets/texture-pack/LICENSE.txt)

Inspired by MC-Bench and VoxelBench