AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

DOOM

doom-checkpoint-state-tracking

You are given one silent first-person Doom gameplay video:

/workspace/materials/doom-checkpoint-state-tracking.mp4

It contains six episodes separated by two-second black gaps. Reconstruct every scored interaction in order and the complete player and world state immediately after it.

Family
Game-state reconstruction
Inputs
video
Media
1 video, 44 min, no sound track
Reference answer
138 events
Deliverable
/workspace/output/solution.json
Scoring
per-componentchecklist / accuracyweightedF1 / precision–recallgated productmean of components

Task package · verifier code

Instruction

Full instruction given to the agent

Doom gameplay event and state reconstruction

You are given one silent first-person Doom gameplay video:

/workspace/materials/doom-checkpoint-state-tracking.mp4

It contains six episodes separated by two-second black gaps. Reconstruct every scored interaction in order and the complete player and world state immediately after it.

Visual identities

  • Key cards are blue_key, yellow_key, and red_key.
  • Weapons are shotgun, chaingun, rocket_launcher, and plasma_rifle.
  • Switch panels use fixed glyphs: amber circle is switch_amber, cyan triangle is switch_cyan, violet diamond is switch_violet, and white cross is switch_white. A bright glyph is on; a dim glyph is off.
  • The green star is checkpoint_alpha; the blue hexagon is checkpoint_beta.
  • Blue, yellow, and red lock panels are door_blue, door_yellow, and door_red.
  • The cyan-violet portal is episode_exit.

Health, ammo, and armor pickups not listed above are unscored distractors. Looking at an object without collecting or activating it is unscored. Failed door and switch uses are unscored.

Scored events

  • key_pickup: first frame after the key sprite is collected.
  • weapon_pickup: first frame on which the collected weapon becomes active.
  • switch_activate: first frame after a successful off-to-on switch use.
  • checkpoint_activate: first frame after a checkpoint becomes current.
  • locked_door_open: first frame on which a keyed door begins opening.
  • checkpoint_restore: first restored frame showing the arrival effect.
  • level_exit: first frame of the exit transition.

The event entity is the collected key or weapon, activated switch or checkpoint, opened door, current checkpoint for a restore, or episode_exit for an exit.

Each episode begins with pistol, no keys, no active switches, no open doors, and a null checkpoint. Keys are not consumed. Open doors never close. Key and weapon pickups, doors, and checkpoints score at most once per episode. A switch can score again only after a checkpoint restore has reverted it to off.

Activating a checkpoint saves the complete active-switch set. On restore, held keys, the active weapon, open doors, and the current checkpoint persist, while active switches revert to the saved set. The exit is the final event and keeps the state visible at transition start.

Timestamps are non-negative integer milliseconds from the first non-black frame of that episode. They must be non-decreasing. Use the first visible event frame; timestamps within 750 ms of that frame are accepted.

Output

Write /workspace/output/solution.json with exactly this shape:

{
  "episodes": [
    {
      "episode_id": "episode_01",
      "events": [
        {
          "timestamp_ms": 12345,
          "event_type": "key_pickup",
          "entity_id": "blue_key",
          "state": {
            "active_weapon": "pistol",
            "held_keys": ["blue_key"],
            "active_switches": [],
            "open_doors": [],
            "current_checkpoint": null
          }
        }
      ]
    }
  ]
}

Episode IDs are episode_01 through episode_06 in video order and each must appear once. State arrays are duplicate-free subsets of the vocabularies above; their order is insignificant. Objects accept exactly the shown keys and no extras.

Stay inside the working directory. Do not use the internet or prior knowledge to fabricate answers; ground every event in the supplied video.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra1.000
Claude Opus 5.50.775
Claude Opus 50.861
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.057

One run per agent.

The agent watches a silent first-person Doom video of six episodes and lists every scored interaction in order (key and weapon pickups, switches, locked doors opening, checkpoints activated and restored, the exit), each with its time and the complete state right after it: active weapon, keys held, switches on, doors open, current checkpoint. The reference holds 23 events per episode, 138 in all.

What a good answer has to get right, and how the verifier checks it

Every interaction is found, in order
Within each episode, entries are aligned with the reference in order. A pair can align only when the event type and the object are the same and the time is within 750 ms.
The full state after each event is right
An aligned event counts in full only when all five state fields are exactly right. This gives the full-event F1 = 2 × full matches / (reference events + reported events).
State is carried forward correctly
State depends on everything before it: keys are kept, doors stay open, and restoring a checkpoint puts the switches back to how they were when it was activated. One early mistake therefore shows up in every later state.
Partly right states earn a little
For aligned events, each of the five state fields is also scored on its own with the same F1 formula, and the five are averaged.

How the checks become one score

Each episode’s score = 0.9 × full-event F1 + 0.1 × mean per-field F1. Score = the mean over the six episodes. The 0.1 part is the only credit for an event found at the right time with a state that is not completely right.

What scores zero

No solution file, one over 1 MB, or one whose top level is not exactly an “episodes” list. An episode that is missing, listed twice, carries extra keys or has more than 100 events scores 0 for that episode. Otherwise, an answer in which no event aligns.

Outside the score: health, ammo and armor pickups, objects looked at but not used, and failed uses of doors and switches, which the task names as distractors. An entry that breaks the event format is dropped before alignment.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
timestamp_tolerance_ms750value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
1.00
Wall time
11.7 min
Steps shown
82

Loading the run…