AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

USC Volleyball

usc-wsu-2023-volleyball-block-timeline

You are given one video at /workspace/materials/match.mp4: the full broadcast of a college women's volleyball match (Southern California at Washington State, five sets). The video has no audio track. Find every point in the match that ended as a block, by either team.

Family
Sports analysis
Inputs
video
Media
1 video, 1 h 58 min, no sound track
Reference answer
23 block points
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallpartial credit

Task package · verifier code

Instruction

Full instruction given to the agent

Block Timeline Reconstruction

You are given one video at /workspace/materials/match.mp4: the full broadcast of a college women's volleyball match (Southern California at Washington State, five sets). The video has no audio track. Find every point in the match that ended as a block, by either team.

For each block point, report the set, the score immediately after the point, the player(s) credited with the block, and the opposing hitter who was blocked. Use any tools in the image (for example ffmpeg and ffprobe) to seek through and sample the video. The on-screen score graphic, the players' jersey numbers, and the play action are your evidence. The rosters below map jersey numbers to names.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "events": [
    {"set": 2, "score_after": "5-9",   "type": "block", "players": ["First Last", "First Last"], "blocked": "First Last"},
    {"set": 4, "score_after": "21-18", "type": "block", "players": ["First Last"], "blocked": "First Last"}
  ]
}
  • One entry per block point, in any order.
  • set: 1 to 5.
  • score_after: the score right after the point, written USC-WashingtonState — the same order the broadcast score graphic displays (USC, the visitor, on the left; Washington State, the home team, on the right).
  • type: always "block" — the point ended at the net off a block (the officially scored block point, not a block touch the attacker still kills through, and not a deflection that stayed in play). Serve aces, attack errors, and other rally endings are not block points and must not be listed.
  • players: the credited blocker or blockers (one or two names, order does not matter). Credit follows the official scorer's rules: a stuff by one player is one name; a block shared by two players at the net is two names.
  • blocked: the opposing hitter who was blocked — the player whose attack the block stopped (one name, on the team opposite the blockers). Required for every entry.

Rosters (jersey number → name)

Southern California (shown left on the score graphic):

# Name # Name
2 London Wijay 13 Tyrah Ariail
4 Megan Verbiest 16 Gala Trubint
5 Skylar Fields 22 Rylie McGinest
6 Ellie Snook 91 Mia Tuaniga
8 Kalyah Williams
10 Lindsey Miller

Washington State (shown right on the score graphic):

# Name # Name
3 Karly Basham 11 Julia Norville
4 Logann Golden 12 Argentina Ung
5 Iman Isanovic 15 Magda Jehlarova
7 Pia Timmer 16 Weronika Wojdyla
8 Katy Ryan 21 Lana Radakovic
9 Shea Rubright

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on memory of this match; find every qualifying point in the video.
  • Report only points that ended as a block: kills (including a kill driven through a block touch), service aces, attack errors, service errors, and other rally endings are not entries.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.277
Claude Opus 5.50.422
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a full college volleyball match and lists every point that ended as a block: the set, the score after the point, the player or players credited with the block, and the opposing hitter who was blocked. The reference holds 23 block points.

What a good answer has to get right, and how the verifier checks it

Every block point is found, and none is invented
An entry is tied to a reference block by the set and the exact score after the point, which is unique within a set. The score is an F1, so missed blocks and entries for points that were not blocks both cost.
The blockers are credited
The set of credited blockers must be exact: a solo block and a shared block are different answers.
The blocked hitter is named
The hitter whose attack was stopped must be right. Blockers and hitter are on opposite sides of the net and have to be read within the same fraction of a second.
One wrong name earns half
An entry whose names are off by exactly one (a blocker missing, extra or wrong, or the hitter wrong) earns 0.5 instead of 1. With two or more errors it earns nothing. Exact matches are assigned first.

How the checks become one score

Score = F1 over credit: precision = credit / entries listed, recall = credit / 23, where credit is 1 per fully right block and 0.5 per block with one name wrong. A last name alone is accepted where it is unambiguous.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry has the right set and score with at most one name wrong.

Outside the score: points won in other ways, such as service aces. The verifier also reports how many block points were found regardless of attribution, as a diagnostic.

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.28
Wall time
13.5 min
Steps shown
104

Loading the run…