AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

Dota 2

dota2-pgl-wallachia-s7-game5-predeath-trajectories

/workspace/materials/final.mp4 is a continuous segment of Game 5 of the PGL Wallachia Season 7 Dota 2 grand final between Team Yandex and Team Liquid. Report every live hero death that belongs to a major teamfight from HUD clock 00:00 through 50:30, inclusive.

Family
Game-state reconstruction
Inputs
video
Media
1 video, 52 min, no sound track
Reference answer
39 deaths
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallgeometric (IoU / distance)

Task package · verifier code

Instruction

Full instruction given to the agent

Task: reconstruct every pre-death minimap trajectory

/workspace/materials/final.mp4 is a continuous segment of Game 5 of the PGL Wallachia Season 7 Dota 2 grand final between Team Yandex and Team Liquid. Report every live hero death that belongs to a major teamfight from HUD clock 00:00 through 50:30, inclusive.

A major teamfight is a maximal sequence containing at least three hero deaths where every death after the first occurs less than 15 seconds after the preceding death. A gap of 15 seconds or more ends the sequence. Do not report isolated deaths or two-death sequences.

For each death, report the HUD clock, the victim, the roster player credited in the live kill feed, and the victim's minimap cell at three times: exactly 10 seconds before the death, exactly 5 seconds before the death, and at the death. death_cell means the victim's final live minimap position immediately before the marker disappears. A broadcast replay or a kill-feed line that remains on screen is not a new death.

Use these canonical player labels:

  • Team Yandex (Radiant): watson, CHIRA_JUNIOR, DM, Malady, Saksa
  • Team Liquid (Dire): tOfu, Ace, Nisha, Boxi, m1CKe

Treat the full square minimap as a 14-by-14 grid. Columns A through N run left to right (west to east), and rows 1 through 14 run bottom to top (south to north). A1 is the Radiant/base corner and N14 is the Dire/base corner. The cell containing the center of the victim's minimap marker is the reported cell. If that center lies exactly on an internal vertical boundary, use the cell to its right (east); on an internal horizontal boundary, use the cell above it (north). The outer east and north edges remain in column N and row 14.

Write the events in chronological order to /workspace/output/solution.json using this exact schema:

{
  "events": [
    {
      "game": 5,
      "clock": "12:34",
      "victim": "Nisha",
      "killer": "watson",
      "cell_10s_before": "E7",
      "cell_5s_before": "F7",
      "death_cell": "G8"
    }
  ]
}

The object above illustrates the format only. Do not add prose or extra fields. Use only the supplied video; do not look up the match, video, event logs, or results online.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.179
Claude Opus 5.50.026
Claude Opus 50.000
Gemini 3.8 Flash0.028
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches the first 50 minutes of a professional Dota 2 game and reports every hero death that belongs to a major teamfight (three or more deaths less than 15 s apart): the clock time, the victim, the player credited with the kill, and the victim’s minimap cell 10 s before the death, 5 s before, and at the death, on a 14 by 14 grid. The reference holds 39 deaths.

What a good answer has to get right, and how the verifier checks it

Every teamfight death is found, and none is invented
The score is an F1 over deaths with one-to-one matching. Isolated deaths and two-death exchanges are not part of the reference, so listing them lowers precision.
The death is timed
The clock time must be within 2 s.
Victim and killer are named
Both must be exact, out of the ten players in the game.
The path before the death is traced
All three minimap cells must be exactly right. A cell is one fourteenth of the map’s width, and the positions at 10 s and 5 s before have to be read from the minimap before the death happens.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 39 deaths and precision over every entry in the file. Every death counts the same, and a death counts only when time, victim, killer and all three cells are right. One cell off by one square makes the entry wrong.

What scores zero

No solution file or one that cannot be read, more than 120 entries, an empty list, or a list in which no entry is right on every field.

Outside the score: deaths outside major teamfights and after 50:30. The verifier also reports F1 for partial matches (time only; time and victim; each cell; all cells within one square) as diagnostics that do not enter the score.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
CLOCK_TOLERANCE_S2constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.18
Wall time
17.6 min
Steps shown
73

Loading the run…