AgenticVBench-Omni

AgenticVBench-Omni / Game-state reconstruction

Melee

melee-nouns-bowl-2025-combo-kill-causal-ledger

The file /workspace/materials/match.mp4 contains ten complete Super Smash Bros. Melee games from three Nouns Bowl 2025 sets, concatenated in this order:

Family
Game-state reconstruction
Inputs
video
Media
1 video, 25 min, no sound track
Reference answer
104 conversions
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

Task: build the Melee combo-and-kill causal ledger

The file /workspace/materials/match.mp4 contains ten complete Super Smash Bros. Melee games from three Nouns Bowl 2025 sets, concatenated in this order:

global game set players
1-3 Ferriswheel vs Zain Ferriswheel, Zain
4-6 JoJo vs Bard JoJo, Bard
7-10 Axe vs SRM13 Axe, SRM13

Reconstruct every qualifying conversion in chronological order. A conversion is one player's continuous punish sequence against the other player.

Operational definitions

  • A conversion starts when a hit or grab puts a player into damage, capture, or command-grab state.
  • It remains the same conversion across subsequent hits and grabs. It ends on stock loss, or after the victim has been in an actionable grounded state for more than 45 consecutive frames without being damaged or grabbed again.
  • hit_count is the number of damage-producing contacts in the conversion. Count each pummel and each damaging part of a multi-hit move. A throw that deals damage counts as a hit.
  • Report a conversion if hit_count >= 4, or if it kills at any hit count.
  • victim_stock_before is the victim's stock count when the conversion starts.
  • Sum the damage added by the conversion and assign damage_band: light for less than 25%, heavy for 25% to less than 50%, and devastating for at least 50%.
  • terminal is kill when the victim loses a stock. It is reversal when the victim starts an overlapping conversion against the original attacker before the original conversion's 45-frame reset completes. Otherwise it is escape.
  • Trades can produce overlapping conversions. Record each qualifying conversion at its own start position in the ledger.
  • Ignore unfinished, non-kill conversions cut off by the end of a game.

Do not look up the sets, replay files, statistics, or results online. Derive the ledger from the supplied video. You may use local video-analysis tools and write intermediate files under /workspace/work.

Write /workspace/output/solution.json with exactly this shape:

{
  "events": [
    {
      "game": 1,
      "attacker": "Zain",
      "victim_stock_before": 4,
      "hit_count": 6,
      "damage_band": "heavy",
      "terminal": "escape"
    }
  ]
}

Use only player tags shown in the table, global game numbers 1 through 10, damage_band values light, heavy, devastating, and terminal values escape, reversal, kill. Do not include timestamps, move names, commentary, confidence scores, or non-qualifying conversions.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.393
Claude Opus 5.50.431
Claude Opus 50.000
Gemini 3.8 Flash0.308
DeepSeek 4.1 Flash0.172

One run per agent.

The agent watches ten complete Super Smash Bros. Melee games from three tournament sets in one video and lists, in order, every conversion: one player’s continuous punish on the other that lands four or more hits or takes a stock. For each it gives the game, the attacker, the victim’s stock count beforehand, the number of hits, a damage band and how it ended. The reference holds 104 conversions.

What a good answer has to get right, and how the verifier checks it

Every conversion is found, in order
The answer is compared with the reference as a sequence: the verifier counts the longest run of entries that appear in both lists in the same order. No timestamps are asked for; the order carries the timing.
The conversion is attributed
Game number, attacker and the victim’s stock count before it must be exact.
The hits are counted
The hit count must be exact. The task defines what counts as a hit, including each pummel and each damaging part of a multi-hit move.
The damage and the ending are classed
The damage band (under 25%, 25% to under 50%, 50% or more) and the ending (kill, reversal or escape) must be exact.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall). Recall is matched conversions over 104; precision is matched conversions over every entry in the file, including entries the verifier rejects as invalid. An entry counts only when all six fields are right.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry is right on all six fields.

Outside the score: conversions of fewer than four hits that do not kill, which the task says not to report; an entry for one is invalid and lowers precision. The verifier also reports matches on game, attacker, stock and ending alone, as a diagnostic.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.43
Wall time
45.5 min
Steps shown
868

Loading the run…