AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

Lacrosse

lacrosse-mich-jhu-2024

You are given one silent video: game.mp4 — the full broadcast of one men's field-lacrosse game (four quarters). The on-screen scoreboard/graphic has been blacked out, and there is no audio. You must reconstruct what happened from the play on the field alone.

Family
Sports analysis
Inputs
videodocuments
Media
1 video, 1 h 42 min, no sound track
Reference answer
26 goals
Deliverable
/workspace/output/solution.json
Scoring
partial creditF1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Task: reconstruct the scoring ledger of a lacrosse game

You are given one silent video: game.mp4 — the full broadcast of one men's field-lacrosse game (four quarters). The on-screen scoreboard/graphic has been blacked out, and there is no audio. You must reconstruct what happened from the play on the field alone.

There are two teams, distinguished by jersey color: - NAVY — dark navy-blue jerseys (numbers in yellow/maize). - WHITE — white jerseys (numbers in light blue).

A closed list of the valid jersey numbers for each team is in roster.json (numbers only — no names are needed or provided). Every number you output must be a number a player actually wears on that team.

What to produce

Write output/solution.json with two ordered lists.

1. goals — every goal, in the order they were scored

For each goal: - team: "NAVY" or "WHITE" — the team that scored. - scorer: the jersey number of the player who scored (integer). - assister: the jersey number of the teammate who threw the pass that led directly to the goal, or null if the goal was unassisted (the scorer created it himself with no immediate assisting pass).

List the goals in chronological order. There is no need to report a clock time.

2. penalties — every penalty, in order

For each penalty: - team: "NAVY" or "WHITE" — the penalized team. - offender: the jersey number of the penalized player, or null if it is a team penalty with no single player (e.g. a face-off violation or offside on the unit). - type: a short lowercase string, one of: "pushing", "slashing", "cross-check", "holding", "unnecessary roughness", "offside", "faceoff violation", "illegal body check", "tripping", "illegal procedure", "other".

Output format

{
  "goals": [
    {"team": "NAVY", "scorer": 89, "assister": null},
    {"team": "WHITE", "scorer": 51, "assister": null},
    {"team": "NAVY", "scorer": 2, "assister": 40}
  ],
  "penalties": [
    {"team": "NAVY", "offender": 5, "type": "pushing"}
  ]
}

Notes

  • A goal counts when the ball fully crosses the goal line into the net in live play (the scoring team celebrates and possession resets to the other team's goalie via a whistle/restart). Do not count shots that are saved, hit the post, or are waved off.
  • Jersey numbers repeat across the two teams (both teams may have a player wearing the same number), so always pair the number with the correct team color.
  • The goal ledger is graded as a full ordered reconstruction: each goal is checked on its team, its scorer, whether it was assisted (assister non-null vs null), AND the running score after it (the cumulative NAVY-WHITE tally, derived from your ordered list). So finding every goal, in the correct order, attributed to the correct team, matters as much as reading the scorer — a missed, extra, or mis-teamed goal shifts the running score for every goal after it. Be exhaustive and get the sequence right. The specific assister number and the penalties are recorded as diagnostics and do not affect the goal score.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.577
Claude Opus 5.50.148
Claude Opus 50.074
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches the full broadcast of a men’s lacrosse game, silent and with the scoreboard blacked out, and lists every goal in order: the team, the scorer’s jersey number, and the assister’s number or none. The reference holds 26 goals.

What a good answer has to get right, and how the verifier checks it

Every goal is found, in order
The answer is compared with the reference as a sequence: the verifier counts the longest run of goals that appear in both lists in the same order.
The team and the scorer are right
The team, read from jersey colour, and the scorer’s number must both be exact. Numbers repeat across the two teams, so the number alone is not enough.
Assisted or not
The entry must say correctly whether the goal was assisted. The assister’s number itself is not required: an audit found it unreadable in the video for about half of the assisted goals.
The running score is right
Each goal also carries the score after it, computed from the answer’s own sequence of teams. A missed or invented goal therefore makes every later goal of that stretch wrong until the count is back in step.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 26 goals and precision over the goals listed. A goal counts only when team, scorer, assisted-or-not and running score are all right.

What scores zero

No answer or one that cannot be read, an empty list, or a list in which no goal is right on all four items.

Outside the score: the assister’s number and the penalties, which the verifier reports as diagnostics without adding them to the score.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.58
Wall time
10.2 min
Steps shown
81

Loading the run…