AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

StarCraft II

sc2-build-order-reconstruction

You are given NINE videos: a 3×3 tiling of ONE bird's-eye view of the WHOLE map of a 1v1 StarCraft II match (Terran vs Zerg), the full ~15-minute game. The tiles are at /workspace/materials/tile_r{row}c{col}.mp4 (row 0=top..2=bottom, col 0=left..2=right) together cover the entire battlefield with no gaps and no clipping. BOTH bases are in the grid: the Terran main (grey/red metal) sits in the right-hand column and the Zerg main (buildings on purple creep) in the left-hand column; each main straddles a tile seam, and both players expand to further bases during the game, so locate the bases yourself instead of assuming one tile per player.

Family
Game-state reconstruction
Inputs
videodocuments
Media
9 videos, 50 min, no sound track
Reference answer
94 build events
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Reconstruct BOTH players' build orders from a tiled bird's-eye StarCraft II video

You are given NINE videos: a 3×3 tiling of ONE bird's-eye view of the WHOLE map of a 1v1 StarCraft II match (Terran vs Zerg), the full ~15-minute game. The tiles are at /workspace/materials/tile_r{row}c{col}.mp4 (row 0=top..2=bottom, col 0=left..2=right) together cover the entire battlefield with no gaps and no clipping. BOTH bases are in the grid: the Terran main (grey/red metal) sits in the right-hand column and the Zerg main (buildings on purple creep) in the left-hand column; each main straddles a tile seam, and both players expand to further bases during the game, so locate the bases yourself instead of assuming one tile per player.

  • The frames are the RAW render — no brightness/contrast processing (so they are dim; you may brighten/adjust them yourself with ffmpeg if it helps you read structures).
  • Each video is 5000 frames sampled ~every 0.18 game-seconds. /workspace/materials/frames_time.json maps frame index → game-seconds, so you can time each event precisely (to ~1 s).
  • The tiles are encoded at 15 fps, so frame i is at i/15 seconds of video time; convert to game-seconds through frames_time.json, never by reading the video clock directly.
  • Time each structure by the moment its construction STARTS — the first frame at which its foundation/pit is visible on the ground, not when it finishes.
  • Buildings do NOT simply persist: they can be destroyed and rebuilt (a rebuild on the same spot is a new event), and a structure's sprite changes without a new building being made (SupplyDepots raise and lower repeatedly, a CommandCenter grows an Orbital dish, a Hatchery becomes a Lair, tanks siege). Judge from the ground footprint, not from any single sprite.
  • There is no HUD, production panel, resource/supply counter, minimap, or player name anywhere in the video — only the rendered game world.

What to produce

Reconstruct, per player, the chronological list of every structure and tech building made, with the game-time (in seconds) each was started. Report EVERY instance — a rebuilt or duplicate building at a new time is a SEPARATE event (do not de-duplicate to a type set).

  • Structure tech morphs ARE separate events, timed at the morph: CommandCenter→OrbitalCommand (or →PlanetaryFortress), Hatchery→Lair→Hive, Spire→GreaterSpire.
  • Add-ons (BarracksTechLab/Reactor, FactoryTechLab/Reactor, StarportTechLab/Reactor) are separate events, timed when the add-on starts.
  • A placement that is CANCELLED before it finishes is NOT an event.
  • Excluded, even though you will see them: units, creep tumors, mineral fields, vespene geysers, destructible rocks, and the two summonables that are not build-order structures (Raven AutoTurret, Reaper KD8Charge).

Scoring: both players are pooled into one timeline; a reported event counts only if the race, the structure name and the time all match a ground-truth event, with |Δt| ≤ 3 seconds. Wrong name, wrong race or a mis-timed event scores nothing, so guessing does not pay.

Structure names — Terran: CommandCenter, OrbitalCommand, PlanetaryFortress, SupplyDepot, Barracks, Refinery, Factory, Starport, EngineeringBay, Bunker, MissileTurret, SensorTower, GhostAcademy, Armory, FusionCore, BarracksTechLab, BarracksReactor, FactoryTechLab, FactoryReactor, StarportTechLab, StarportReactor. Zerg: Hatchery, Lair, Hive, SpawningPool, Extractor, RoachWarren, BanelingNest, EvolutionChamber, HydraliskDen, InfestationPit, Spire, GreaterSpire, NydusNetwork, UltraliskCavern, SpineCrawler, SporeCrawler. (Name matching is case/space-insensitive.)

Output → write /workspace/output/solution.json

The JSON below shows the required format only — its names and times are invented placeholders, not events from this match.

{"players":[
  {"race":"terran","buildings":[{"t_seconds":137,"name":"Factory"},{"t_seconds":642,"name":"Armory"}]},
  {"race":"zerg","buildings":[{"t_seconds":262,"name":"RoachWarren"},{"t_seconds":708,"name":"SpineCrawler"}]}
]}

Watch the entire game across all 9 tiles, using every tool available to you over many rounds of tool calls (50+ preferred). Find each base among the tiles, track construction starts, and time each via /workspace/materials/frames_time.json. Write /workspace/output/solution.json early and refine it; do not fabricate. The nine tiles + frames_time.json are the only inputs provided.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.768
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.173
DeepSeek 4.1 Flash0.248

One run per agent.

The agent gets nine videos that tile a bird’s-eye view of the whole map of a 15-minute StarCraft II match, Terran against Zerg. It lists every structure each player builds and when. The reference holds 94 build events: 61 Terran and 33 Zerg.

What a good answer has to get right, and how the verifier checks it

Every structure is found, and none is invented
The score is an F1 over build events of both players pooled into one timeline; each reference event can be matched once, by the closest entry in time.
The structure is named
The race and the structure name must both be exact, compared without regard to capitals or spaces.
The build is timed
The time must be within 3 s in game time. Times may be given in seconds or as minutes and seconds.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 94 events. Every event counts the same. The two players are not scored separately, so the Terran player’s 61 events carry about two thirds of the weight.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry has the right race and structure within 3 s.

Outside the score: units, upgrades and anything other than structures. Two layouts of the answer file are accepted and scored the same.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
tol_s3value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.77
Wall time
11.2 min
Steps shown
67

Loading the run…