AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

Balance Beam

pac12-2018-balance-beam-routine-timeline

You are given /workspace/materials/source.mp4, a full gymnastics competition broadcast that interleaves vault, uneven-bars, balance-beam, and floor-exercise performances.

Family
Sports analysis
Inputs
videodocuments
Media
1 video, 2 h 19 min, with sound
Reference answer
23 routines
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Balance-Beam Routine Timeline, Dismount Takeoff, Gymnast, and Score

You are given /workspace/materials/source.mp4, a full gymnastics competition broadcast that interleaves vault, uneven-bars, balance-beam, and floor-exercise performances.

Find every complete balance-beam routine shown as the live foreground performance. For each routine, report its mount start, final dismount takeoff, dismount landing, the school the gymnast represents, the gymnast's name, and the official beam score shown or made derivable by the broadcast for that routine. Also report the first video timestamp at which enough score information is readable to determine that individual score.

Use the video as the source of truth. You may use tools available in the image, including ffmpeg, ffprobe, and Python, to seek, extract, and inspect frames.

Submission

Write /workspace/output/solution.json with exactly this structure:

{
  "beam_routines": [
    {
      "start_time": "HH:MM:SS.mmm",
      "end_time": "HH:MM:SS.mmm",
      "dismount_takeoff_time": "HH:MM:SS.mmm",
      "school": "<allowed school label>",
      "gymnast_name": "<full broadcast name>",
      "beam_score": "D.DDD",
      "score_time": "HH:MM:SS.mmm"
    }
  ]
}

Sort entries strictly chronologically by start_time. Use exactly seven fields per entry. Any malformed record, extra field, or out-of-order entry invalidates the whole submission.

All four timestamps must use zero-padded HH:MM:SS.mmm video timestamps measured from the beginning of the file.

Use exactly one of these school labels:

  • Stanford
  • Arizona State
  • Oregon State
  • Arizona

Identify the school from the supplied broadcast's visible team, athlete, and on-screen context. The school must be correct for a routine to match.

Use the gymnast's full name as displayed or announced by the supplied broadcast. gymnast_name must be a non-empty string with no leading or trailing whitespace and must match the routine's gymnast exactly.

Use the official individual beam score directly shown for that routine, except for the sixth-gymnast inference described below. Do not substitute a team total or another apparatus score. Write beam_score as a string with exactly three digits after the decimal point, including trailing zeroes such as "9.800" and "9.850". The score must match exactly for the record to match. The score graphic may appear after the dismount or during later coverage, so track the gymnast-routine association across intervening broadcast cuts.

Write score_time as a zero-padded HH:MM:SS.mmm timestamp. For a directly shown individual score, use the first frame on which the gymnast's official score is readable. The score timestamp is scored with a tight ±1.00-second tolerance.

The sixth beam gymnast for each school does not receive a later individual score graphic in this broadcast. Infer that gymnast's score from the completed beam-rotation subtotal shown in the later standings table. A team beam subtotal is the sum of its best five scores from the six-person beam lineup. Track all six lineup scores needed for this calculation, including a score belonging to a performance omitted from the output because its mount is not visible. In this broadcast, each sixth score replaces the lowest of the first five; use:

sixth score = completed beam subtotal - sum(first five scores) + lowest(first five scores)

For an inferred sixth score, score_time is the first frame on which that school's completed beam subtotal is readable in the standings table. Do not submit the team subtotal itself as beam_score.

Routine boundaries

  • Start at the first visible frame of takeoff or intentional weight transfer into the mount. Exclude the salute and passive approach. This timestamp is scored with a ±0.25-second tolerance.
  • Set dismount_takeoff_time to the first frame after the gymnast's final supporting foot or toe loses contact with the beam for the dismount. For a connected roundoff or back-handspring dismount, use the final foot rebound that launches the airborne dismount, not the earlier entry into hand support. This timestamp must fall between start_time and end_time and is scored with a ±0.25-second tolerance.
  • End at the first frame where the gymnast's feet contact the landing mat after the final dismount. This timestamp is scored with a ±0.25-second tolerance.
  • Count only distinct, live, foreground routines whose mounts and final landings are visible and whose performances are sufficiently complete to evaluate.
  • Do not count replays, warm-ups, or incomplete background performances as separate routines.

Constraints

  • Stay inside /workspace.
  • Do not use the internet, web search, or outside knowledge. Do not rely on memory of this competition; derive every submitted record from the supplied video.
  • Do not include commentary or evidence fields in the JSON.
  • You have one non-extendable 90-minute run for all inspection and writing. Checkpoints do not pause or extend that deadline, and the task must not be resumed after it expires. Submit the best valid solution before the deadline.

Progress checkpoint

During the single run, after each substantial scan segment, update /workspace/work/checkpoint.json with valid JSON containing:

  • the last fully scanned video timestamp;
  • candidate routine windows found so far;
  • which candidates still need boundary review; and
  • the next concrete inspection step.

The checkpoint is operational only and is not part of the scored submission.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.087
Claude Opus 5.50.128
Claude Opus 50.170
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a full gymnastics meet broadcast in which vault, bars, beam and floor are interleaved, and lists every complete balance-beam routine: when it starts and ends, the moment of takeoff for the dismount, the school, the gymnast’s name, the score shown and when it appears. The reference holds 23 routines.

What a good answer has to get right, and how the verifier checks it

Every beam routine is found, and none is invented
The score is an F1 over routines with one-to-one matching.
The routine is bounded
Start and end must each be within 0.25 s.
The dismount takeoff is located
The moment the gymnast leaves the beam for the dismount must be within 0.25 s.
The gymnast is identified
School and gymnast name must be exact.
The score is read
The beam score must be exact to three decimals, and the time it appears within 1.0 s.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 23 routines. Every routine counts the same, and a routine counts only when all seven fields are right.

What scores zero

No solution file or one that cannot be read. Also any single record that breaks the format: a missing or extra field, a time not in the required form, a school outside the four competing, times out of order within a record, or records not in order of start time. The instruction states that one such record invalidates the whole submission. Otherwise, a list in which no routine is right on all seven fields.

Outside the score: routines on the other three apparatus, and beam routines that are not shown complete as the live performance. The verifier also reports looser matches as diagnostics.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
START_TOLERANCE_S0.25constant in the verifier code
END_TOLERANCE_S0.25constant in the verifier code
DISMOUNT_TAKEOFF_TOLERANCE_S0.25constant in the verifier code
SCORE_TIME_TOLERANCE_S1constant in the verifier code
DIAGNOSTIC_MATCH_WINDOW_S20constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5 · Claude Code 2.1.269
Score
0.17
Wall time
33 min
Steps shown
388

Loading the run…