AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

LoL

lol-minimap-timeline-reconstruction

You are given one video at /workspace/materials/minimap_vod.mp4: the map view only of one full League of Legends game (spectator, full-map vision), recorded real time from game start 0:00. There is no main camera, HUD, clock, killfeed, scoreboard, names, gold, or objective-spawn timers — only the minimap. An event's game_clock_s is its elapsed seconds in the video (read frame time from the video itself; no on-screen clock is shown).

Family
Game-state reconstruction
Inputs
video
Media
1 video, 40 min, no sound track
Reference answer
111 events
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

LoL Minimap Timeline + Economy Reconstruction

You are given one video at /workspace/materials/minimap_vod.mp4: the map view only of one full League of Legends game (spectator, full-map vision), recorded real time from game start 0:00. There is no main camera, HUD, clock, killfeed, scoreboard, names, gold, or objective-spawn timers — only the minimap. An event's game_clock_s is its elapsed seconds in the video (read frame time from the video itself; no on-screen clock is shown).

Reconstruct every key event of the game in chronological order. Each event is a 5-tuple (game_clock_s, type, entity, minute_gain, leader_before).

  • type: one of champion_kill | tower_kill | epic_monster_kill
  • game_clock_s: integer seconds (elapsed video time)
  • entity depends on type:
  • champion_kill → the dead champion's official English name exactly as written on leagueoflegends.com/champions, including apostrophes and capitalization (e.g. Ahri, Kai'Sa, Cho'Gath). Matching ignores case only.
  • tower_kill → a TURRET only (ignore inhibitors). "{team}_{lane}_{tier}", where team is blue|red (the side that destroyed it), lane is top|mid|bot, or base for the two Nexus turrets, and tier is outer|inner|inhib|nexus. The two Nexus turrets are both "{team}_base_nexus" — do not distinguish left/right. Examples: red_top_outer, blue_base_nexus.
  • epic_monster_kill → the specific monster: a drake element infernal|ocean|mountain|cloud|hextech|chemtech, or elder (elder dragon), or baron (dragons and Baron are the SAME type; the entity says which).
  • minute_gain: blue|red|equal — which side gained more gold (≥300 net) during the game-minute containing the event, else equal.
  • leader_before: blue|red|equal — which side led (by ≥1000 gold) at the whole minute just before the event, else equal.

On the map: champion icons are portraits (identity is readable), structures are fixed icons that vanish permanently when destroyed, and vision is full-map, so a champion icon disappearing is a death (it reappears at the fountain after the respawn timer), not fog. Blue base is bottom-left, red base is top-right. No gold is ever shown — the two economy fields must be inferred from map macro (who took the kills/towers/objectives, who held wave and map pressure, the cumulative state of towers/control/vision up to that point).

IGNORE inhibitors: do NOT report inhibitor structures at all. Only turrets are tower_kill events.

What to submit

Write /workspace/output/solution.json in exactly this shape (the example values are made-up, not from this game):

{
  "events": [
    {"game_clock_s": 196, "type": "champion_kill", "entity": "Taliyah", "minute_gain": "blue", "leader_before": "equal"},
    {"game_clock_s": 380, "type": "epic_monster_kill", "entity": "chemtech", "minute_gain": "equal", "leader_before": "blue"},
    {"game_clock_s": 826, "type": "tower_kill", "entity": "red_top_outer", "minute_gain": "red", "leader_before": "blue"}
  ]
}
  • One entry per key event, in any order.
  • This game has 111 events: 86 champion_kill, 16 tower_kill, 9 epic_monster_kill (6 drakes, 3 barons). Blue: Garen, Zac, Ahri, Kai'Sa, Thresh; red: Vayne, Sejuani, Yone, Taliyah, Neeko.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on memory of any game; find every event in the video. This is a private game with no public timeline.
  • Watch the entire video carefully, using every tool available to you (ffmpeg, ffprobe, frame extraction, etc.). Do not stop early — confirm you have covered the whole game before you answer.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.279
Claude Opus 5.50.126
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.110

One run per agent.

The agent gets the minimap view of one full League of Legends game, about 39.5 minutes, with no clock, kill feed, names or gold shown. It lists every key event: its time, its type (champion kill, tower kill, epic monster kill), who or what it concerns, and two judgements about gold at that moment. The reference holds 111 events.

What a good answer has to get right, and how the verifier checks it

Every key event is found, and none is invented
The score is an F1 over events; each reference event can be matched once, by the closest entry in time. A missed event lowers recall and an entry that matches nothing lowers precision.
The event is timed
The time must be within 3 s of the true game clock, read from the video’s own frame time.
The event is identified
Type and entity must both be exact: which champion died, which of the towers fell, which dragon or baron was taken. Champions appear only as small portraits on the map.
The state of the economy is judged
Two further fields must be exact: which side gained more gold during that minute (by 300 or more), and which side was ahead before it (by 1,000 or more), each blue, red or equal. Gold is never shown, so both have to be inferred from what happens on the map.
A player can make that judgement from the map
In a check made during development, two players ranked Emerald were given only the video and instruction that the agent gets. Each got about 85% of each gold field right, and their answers averaged 0.79 under this verifier. Their misses were mostly early in the game, where the difference between the sides is close to the thresholds.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 111 events. Every event counts the same, and an event counts only when time, type, entity and both gold fields are right.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry is right on every field.

Scores from this verifier: the reference answer with one thing changed

SubmissionScorePrecisionRecall
Careful work
The reference answer1.0001.0001.000
Every time 2 s offEach time is reported 2 s late or early, alternately.1.0001.0001.000
One kind of mistake
Champion kills onlyThe 25 tower and epic-monster events are left out. Everything else is correct.0.8731.0000.775
Kills found, champion wrongEvery champion kill names the champion of the next kill instead. Towers and monsters are correct.0.3960.3960.396
Gold leader always blueThe side in the lead is given as blue for every event. Everything else is correct.0.3600.3600.360
Gold never judgedBoth gold fields are given as “equal” for every event. Times, types and entities are correct.0.0540.0540.054
Every time 5 s lateEverything else is correct.0.0090.0090.009
No real work
Empty list0.0000.0000.000
No file0.0000.0000.000
An agent’s answer
A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step0.2790.2790.279

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: nothing in the answer is ignored. Answers in an earlier field layout are converted before scoring.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
tolerance_s3value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.28
Wall time
11.7 min
Steps shown
58

Loading the run…