AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

GSW

gsw-cle-2018-finals-g4-three-point-timeline

You are given one video at /workspace/materials/game.mp4: the full broadcast of an NBA game (Golden State Warriors vs Cleveland Cavaliers). Reconstruct the complete timeline of every made three-point field goal in the game, by either team.

Family
Sports analysis
Inputs
video
Media
1 video, 2 h 35 min, with sound
Reference answer
22 made three-pointers
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallpartial credit

Task package · verifier code

Instruction

Full instruction given to the agent

Three-Point Timeline Reconstruction

You are given one video at /workspace/materials/game.mp4: the full broadcast of an NBA game (Golden State Warriors vs Cleveland Cavaliers). Reconstruct the complete timeline of every made three-point field goal in the game, by either team.

For each made three, report the quarter, the game clock at the moment it was made, who made it, and who assisted it (or unassisted). Use any tools in the image (for example ffmpeg and ffprobe) to seek through and sample the video. The on-screen score-and-clock graphic, the players' jerseys, and the play action are your evidence.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "three_pointers": [
    {"quarter": 1, "clock": "11:02", "shooter": "JR Smith",      "assister": "LeBron James"},
    {"quarter": 1, "clock": "8:47",  "shooter": "Stephen Curry", "assister": "unassisted"}
  ]
}
  • One entry per made three-pointer, in any order.
  • quarter: 1 to 4.
  • clock: the game clock shown when the shot was made, as on the broadcast (mm:ss, or seconds when under a minute).
  • shooter: the player's full name (first and last).
  • assister: the assisting player's full name, or unassisted.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on memory of this game; find every shot in the video.
  • Count only made three-pointers, not attempts, not two-pointers, not free throws.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.500
Claude Opus 5.50.651
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a full basketball game broadcast, about two and a half hours, and lists every made three-pointer: the quarter, the game clock, the shooter and the player who assisted, or that it was unassisted. The reference holds 22 made threes, 4 of them unassisted.

What a good answer has to get right, and how the verifier checks it

Every made three is found, and none is invented
The score is an F1 over made threes; each reference three can be matched once. Every entry in the list counts as a claim.
The shot is placed in the game
The quarter must be exact and the game clock within 5 s.
The shooter is named
The shooter must be right. A last name alone is accepted where only one shooter in the game has it.
The assist is attributed
The assister must be the right player, or “unassisted” where there was none. This requires watching the pass before the shot.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 22 threes. Every three counts the same, and an entry counts only when quarter, clock, shooter and assister are all right.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry is right on all four fields.

Outside the score: missed three-point attempts and two-point shots. The verifier also reports how many entries have quarter, clock and shooter right regardless of the assister, as a diagnostic.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
clock_tolerance_s5value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.65
Wall time
29.8 min
Steps shown
563

Loading the run…