AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

BYU Volleyball

byu-wsu-2023-volleyball-block-timeline

You are given one video at /workspace/materials/match.mp4: the full broadcast of a college women's volleyball match (BYU at Washington State, four sets). The video has no audio track. Find every point in the match that ended as a block, by either team.

Family
Sports analysis
Inputs
video
Media
1 video, 1 h 55 min, no sound track
Reference answer
18 block points
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallpartial credit

Task package · verifier code

Instruction

Full instruction given to the agent

Block Timeline Reconstruction

You are given one video at /workspace/materials/match.mp4: the full broadcast of a college women's volleyball match (BYU at Washington State, four sets). The video has no audio track. Find every point in the match that ended as a block, by either team.

For each block point, report the set, the score immediately after the point, the player(s) credited with the block, the opposing hitter who was blocked, and the setter who fed that attack. Use any tools in the image (for example ffmpeg and ffprobe) to seek through and sample the video. The on-screen score graphic, the players' jersey numbers, and the play action are your evidence. The rosters below map jersey numbers to names.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "events": [
    {"set": 2, "score_after": "7-12", "type": "block", "players": ["First Last", "First Last"], "blocked": "First Last", "setter": "First Last"},
    {"set": 4, "score_after": "20-16", "type": "block", "players": ["First Last"], "blocked": "First Last", "setter": "First Last"}
  ]
}
  • One entry per block point, in any order.
  • set: 1 to 4.
  • score_after: the score right after the point, written BYU-WashingtonState — the same order the broadcast score graphic displays (BYU on the left).
  • type: always "block" — the point ended at the net off a block (the officially scored block point, not a block touch the attacker still kills through, and not a deflection that stayed in play). Serve aces, attack errors, and other rally endings are not block points and must not be listed.
  • players: the credited blocker or blockers (one or two names, order does not matter). Credit follows the official scorer's rules: a stuff by one player is one name; a block shared by two players at the net is two names.
  • blocked: the opposing hitter who was blocked — the player whose attack the block stopped (one name, on the team opposite the blockers).
  • setter: the player who set that attack — the team-mate who delivered the ball to the blocked hitter earlier in the same rally (one name, same team as blocked).

Rosters (jersey number → name)

BYU (shown left on the score graphic):

# Name # Name
1 Kate Prior 12 Claire Little
4 Hannah Billeter 13 Mia Lee
7 Whitney Bower 14 Aria McComber
8 Eden Bower 15 Elyse Stowell
9 Alyssa Erickson 18 Kamaile Hiapo
10 Erin Livingston 21 Whitney McEwan-Llarenas
24 Brielle Kemavor

Washington State (shown right on the score graphic):

# Name # Name
3 Karly Basham 12 Argentina Ung
5 Iman Isanovic 13 Emma Barbero
7 Pia Timmer 15 Magda Jehlarova
8 Katy Ryan 21 Lana Radakovic
11 Julia Norville

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on memory of this match; find every qualifying point in the video.
  • Report only points that ended as a block: kills (including a kill driven through a block touch), service aces, attack errors, service errors, and other rally endings are not entries.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.250
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.040
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a full college volleyball match and lists every point that ended as a block: the set, the score after the point, the player or players credited with the block, the opposing hitter who was blocked, and the setter who fed that attack. The reference holds 18 block points.

What a good answer has to get right, and how the verifier checks it

Every block point is found, and none is invented
An entry is tied to a reference block by the set and the exact score after the point, which is unique within a set. The score is an F1, so missed blocks and entries for points that were not blocks both cost.
The blockers are credited
The set of credited blockers must be exact: a solo block and a shared block are different answers.
The blocked hitter is named
The hitter whose attack was stopped must be right.
The setter is named
The setter who fed the attack must be right. This has to be followed back through the rally; it cannot be read from the celebration after the point.
One slip in attribution earns half
An entry with exactly one of the three attributions wrong earns 0.5 instead of 1. With two or more wrong it earns nothing. Exact matches are assigned first, so a half-right entry cannot take the place of a fully right one.

How the checks become one score

Score = F1 over credit: precision = credit / entries listed, recall = credit / 18, where credit is 1 per fully right block and 0.5 per block with one attribution wrong. A last name alone is accepted where it is unambiguous.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no entry has the right set and score with at most one attribution wrong.

Outside the score: one further block point, in set 2 at 1-2, whose official record is corrupted. An entry for it is set aside: it earns nothing and costs nothing. The verifier also reports how many block points were found regardless of attribution, as a diagnostic.

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.25
Wall time
16.2 min
Steps shown
151

Loading the run…