AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

Minecraft

minecraft-gameplay-ledger-s1

You are given one video at /workspace/materials/game.mp4: a first-person recording of a Minecraft session. The player travels across eight biomes (forest, beach, desert, snowy tundra, jungle, plains, savanna, badlands), gathers many kinds of block, builds a house block-by-block on camera, hunts animals with a diamond sword and with a bow and arrow, and finally digs a staircase mine and extracts every ore it exposes.

Family
Game-state reconstruction
Inputs
video
Media
1 video, 3 h 58 min, no sound track
Reference answer
1,995 events
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallweightedone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Minecraft First-Person Gameplay Ledger Reconstruction

You are given one video at /workspace/materials/game.mp4: a first-person recording of a Minecraft session. The player travels across eight biomes (forest, beach, desert, snowy tundra, jungle, plains, savanna, badlands), gathers many kinds of block, builds a house block-by-block on camera, hunts animals with a diamond sword and with a bow and arrow, and finally digs a staircase mine and extracts every ore it exposes.

Reconstruct the player's deliberate action ledger in the order the actions happen:

  • mine — the player breaks a block; report the block type.
  • place — the player places a block (while building); report the block type.
  • kill — the player kills a mob; report the mob type and the weapon used.

Identify blocks and mobs from their rendered textures. The weapon is visible two ways: the selected slot in the on-screen hotbar, and how the animal dies — a sword kill happens with the player right next to the animal and swinging, while a bow kill happens at a distance with arrows in flight. Use any tools in the image (for example ffmpeg and ffprobe) to seek through and sample the video.

What to submit

Write /workspace/output/solution.json, actions in chronological order:

{
  "events": [
    {"action": "mine",  "target": "oak_log",      "t": 142.5},
    {"action": "kill",  "target": "cow",   "tool": "sword", "t": 210.0},
    {"action": "place", "target": "stone_bricks", "t": 305.2},
    {"action": "kill",  "target": "chicken", "tool": "bow", "t": 480.7},
    {"action": "mine",  "target": "iron_ore",     "t": 12530.0}
  ]
}
  • action: mine, place, or kill.
  • target: a block type (mine/place) or a mob type (kill), from the vocabulary below.
  • tool: sword or bow — required on kill events, ignored elsewhere.
  • t: the time in the video, in seconds from the start, at which the action happens (a number, e.g. 142.5; a mm:ss / h:mm:ss clock string is also accepted). Required on every event — it does not have to be exact, within ~10 s is enough (see scoring).

Vocabulary (closed)

Blocks (mine/place): oak_log, birch_log, spruce_log, jungle_log, acacia_log, dark_oak_log, oak_leaves, birch_leaves, spruce_leaves, grass_block, dirt, gravel, stone, cobblestone, stone_bricks, andesite, granite, diorite, sand, sandstone, cactus, snow, snow_block, ice, packed_ice, red_sand, terracotta, orange_terracotta, white_terracotta, red_terracotta, yellow_terracotta, brown_terracotta, light_gray_terracotta, red_sandstone, coarse_dirt, podzol, clay, coal_ore, iron_ore, gold_ore, redstone_ore, diamond_ore, lapis_ore, emerald_ore, oak_planks, spruce_planks, birch_planks, jungle_planks, acacia_planks, oak_stairs, spruce_stairs, birch_stairs, jungle_stairs, acacia_stairs, oak_fence, glass, oak_door, torch.

Mobs (kill): cow, pig, sheep, chicken, wolf, mooshroom, polar_bear, turtle, panda.

How it is scored

Your events are matched to the ground truth by (action, target), in order. A predicted event can match a true event only if its time t is within ±10 seconds of the true time — you do not need exact times, but a ledger pinned to the wrong parts of the video will not match.

Reconstruct as much of the ledger as you can, in order: the task is to recover most of what happens across the whole video, not a safe fraction of it, so watch all of it.

Every target is nameable from the closed vocabulary. The block and mob textures are stock Minecraft for exactly the named types in the vocabulary above — nothing is retextured or randomized. Different renders vary which named blocks appear (the palette), never how a block looks, so every on-screen block/mob maps to one vocabulary entry with no legend lookup required.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online. Reconstruct the actions from the video, in order.
  • Report only mine, place, and kill actions (not walking, looking, or pickups).

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.107
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.011
DeepSeek 4.1 Flash0.003

One run per agent.

The agent watches a first-person recording of a Minecraft session, about 238 minutes long, and writes the ordered list of deliberate actions: each block mined, each block placed and each mob killed, with what it was, when, and for a kill the weapon. The reference holds 1,995 events: 1,501 placed, 355 mined and 139 kills.

What a good answer has to get right, and how the verifier checks it

Each action is identified
An entry can only align with a true event that has the same action and the same target, such as mine oak log or kill cow.
Each action is placed in time
Its time must be within 10 s of the true event. The instruction states this window.
The order is kept
Aligned pairs must be in the same order in both lists (a longest common subsequence). One missed, extra or wrong entry costs only itself, but the right events attached to the wrong parts of the video do not align.
The whole ledger is covered
The ledger score is an F2, which weights recall above precision (β = 2), because the task is to reconstruct the whole session. Listing less to stay safe is penalized more than a few extra entries.
The weapon is named for each kill
For kills that were aligned, the weapon, sword or bow, must match. This gives an F1 over the 139 kills; a kill that was not located earns no weapon credit.

How the checks become one score

Score = 0.85 × ledger F2 + 0.15 × weapon F1. F2 = 5 × precision × recall / (4 × precision + recall), with precision and recall counted over aligned events.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry has the right action and target within 10 s.

Outside the score: walking, looking around and anything that is not mining, placing or killing; entries without a readable time cannot align.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
TIME_TOL10constant in the verifier code
time_tol_s10value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.11
Wall time
15 min
Steps shown
72

Loading the run…