AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

OpenTTGames

openttgames-rally-event-chain

Analyze the complete table-tennis match at:

/workspace/materials/game.mp4

Reconstruct the ordered stroke chain and terminal outcome for every live-play rally in the match.

Family
Sports analysis
Inputs
video
Media
1 video, 24 min, no sound track
Reference answer
92 rallies and 387 strokes
Deliverable
/workspace/output
Scoring
F1 / precision–recallgated product

Task package · verifier code

Instruction

Full instruction given to the agent

Full-match table-tennis rally event-chain reconstruction

Analyze the complete table-tennis match at:

/workspace/materials/game.mp4

Reconstruct the ordered stroke chain and terminal outcome for every live-play rally in the match.

Use local tools such as FFmpeg, ffprobe, scripts, and frame extraction. Inspect the full video. Do not use the Internet, pre-existing annotations, or files outside /workspace.

Rally and player conventions

  • A rally begins at the server's racket-ball contact and ends at the first supported terminal event.
  • Count the serve as the first stroke.
  • A stroke is a live-play racket-ball contact intended to serve or return the ball.
  • Do not count warm-ups, dead-ball swings, ball handling between points, or other non-live-play actions.
  • Report only racket-ball contacts you can actually observe in the video. The camera framing is narrower than the playing area, and a player who moves back to return a ball may leave the frame entirely. Do not infer or report a contact that occurs outside the camera frame, even when the ball's later path implies one must have happened. The same applies to a contact whose moment is not visible in the footage.
  • Where a contact is observable but one of its categorical fields is not, do not guess: omit the whole stroke rather than report it with an inferred player, hand, or stroke value.
  • Player identity is determined by fixed image position:
  • left: the player positioned on the left side of the table in the video.
  • right: the player positioned on the right side of the table in the video.
  • Reconstruct every live-play rally in the match whose serve contact you can observe. A rally is anchored on its serve, so if a point is played but its serve's racket-ball contact is not visible in the footage, omit that rally entirely rather than reporting it with an estimated serve time.
  • Report rallies in chronological order by serve time.
  • Report strokes within each rally in chronological order.

Hand classification

hand must be exactly one of:

  • forehand: the player contacts the ball using a forehand stroke orientation, with the racket swung from the racket-hand side of the body.
  • backhand: the player contacts the ball using a backhand stroke orientation, with the racket presented across or in front of the body using a backhand stroke motion.

Classify forehand/backhand from the visible stroke motion, not from whether the player appears on the left or right side of the image.

Stroke technique classification

stroke must be exactly one of:

  • serve: the initial racket-ball contact that starts the rally.
  • loop: an attacking topspin stroke with a pronounced forward/upward brushing swing.
  • block: a compact reactive return that primarily redirects the opponent's pace, usually with little backswing.
  • push: a controlled underspin/backspin return produced with an open racket and a short forward/downward stroke.
  • flick: a quick attacking stroke against a short ball, typically played close to or over the table with a compact wrist/forearm action.
  • lob: a high-arcing defensive return intended to send the ball high and deep.
  • smash: a forceful attacking stroke, usually relatively flat and directed strongly forward/downward against a high or attackable ball.
  • chop: a defensive slicing stroke with a pronounced downward motion that imparts backspin.

Rally-ending convention

Every ending label has the form:

<player>_<ending-category>

The left or right prefix identifies the player whose action or failure causes that terminal outcome.

The ending categories are:

  • net: the prefixed player strikes the ball and it is stopped by the net, or touches the net and clearly returns to that same player's side.
  • not_hitting_ball: the prefixed player is the receiving player who attempts to play the ball but fails to make racket contact.
  • winner: the prefixed player hits a legal shot that becomes unreachable for the opponent.
  • double_bounce: the prefixed player hits a legal shot that bounces twice on the opponent's side before the opponent returns it.
  • out: the prefixed player sends the ball beyond the opponent's side without a legal table bounce.
  • miss_on_own_side: after the prefixed player's racket contact, the ball falls below the table on that same player's side.

The allowed ending labels are exactly:

  • left_out
  • right_out
  • left_net
  • right_net
  • left_winner
  • right_winner
  • left_double_bounce
  • right_double_bounce
  • left_not_hitting_ball
  • right_not_hitting_ball
  • left_miss_on_own_side
  • right_miss_on_own_side
Net-contact rule

A ball touching the net is not automatically a rally ending.

  • If the ball touches the net and still lands legally on the opponent's side, play continues.
  • If the ball is stopped by the net or clearly returns toward the striking player's own side, use <player>_net.
  • If the ball touches the net and then continues out of play, classify the terminal outcome according to where the ball ultimately goes, such as out or miss_on_own_side.

Timestamp anchors

All timestamps are seconds from the first video frame.

  • serve_time_sec: timestamp of the server's racket-ball contact.
  • stroke time_sec: timestamp of that stroke's racket-ball contact.
  • ending_time_sec: timestamp of the observable terminal event using the category-specific rule below:
  • net: when the ball is stopped by the net, or when net contact makes it clear that the ball will return to the striking player's own side.
  • not_hitting_ball: when the receiving player has failed to make racket contact and the ball has passed that player's playable racket position.
  • winner: when the ball becomes unreachable for the receiving player. If the ball contacts or is caught by the receiving player before another clear unreachable point, use that contact as the observable anchor.
  • double_bounce: at the second bounce on the receiving player's side. This is the benchmark's operational timestamp anchor for this category.
  • out: when the ball has fully passed the table boundary on the opponent's side without a legal table bounce.
  • miss_on_own_side: when the ball has fully dipped below the table on the striking player's own side.

Required output

Write valid JSON to:

/workspace/output/solution.json

Use this schema:

{
  "rallies": [
    {
      "serve_time_sec": 5000.000,
      "strokes": [
        {
          "time_sec": 5000.000,
          "player": "right",
          "hand": "forehand",
          "stroke": "serve"
        },
        {
          "time_sec": 5000.700,
          "player": "left",
          "hand": "backhand",
          "stroke": "push"
        }
      ],
      "ending_time_sec": 5001.600,
      "ending": "left_out"
    }
  ]
}

The numeric values in this example are synthetic schema examples only and lie outside the duration of the provided video.

Evaluation tolerances

  • serve_time_sec: within 1.0 second of the serve contact.
  • each stroke time_sec: within 0.35 seconds of the racket contact.
  • ending_time_sec: within 1.0 second of the terminal rally event.

Categorical fields must use exactly the vocabulary defined above.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.334
Claude Opus 5.50.000
Claude Opus 50.028
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.001

One run per agent.

The agent watches a complete table-tennis match and reconstructs every rally: when the serve was struck, every stroke with its time, player, forehand or backhand and stroke type, and how and when the rally ended. The reference holds 92 rallies and 387 strokes.

What a good answer has to get right, and how the verifier checks it

Every rally is found
A reported rally is paired with a reference rally when its serve time is within 1.0 s, one to one. This gives an F1 over rallies.
Every stroke is timed
Within a paired rally, strokes are matched in order when their times are within 0.35 s of the racket contact. This gives an F1 over strokes. Strokes reported in a rally that was not paired count as wrong.
Every stroke is described
A matched stroke counts in the description score only when the player, the hand and the stroke type are all right. This gives a second F1 over strokes.
The ending is right
For each paired rally, the ending label must be right and the ending time within 1.0 s. The ending score is the share of paired rallies where both hold.

How the checks become one score

Score = rally F1 × ending score × stroke timing F1 × stroke description F1. The four parts are multiplied, so the score cannot be higher than the weakest part: an answer that times every stroke and names half of them correctly scores at most one half.

What scores zero

No solution file, one over 8 MB or one that cannot be read, or any of the four parts at zero: no rally paired, no ending right, no stroke timed, or no timed stroke fully described.

Outside the score: anything between rallies. The verifier also reports each stroke field and the ending label and time separately, as diagnostics.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
RALLY_START_TOLERANCE_S1constant in the verifier code
STROKE_TOLERANCE_S0.35constant in the verifier code
ENDING_TOLERANCE_S1constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.33
Wall time
25.8 min
Steps shown
161

Loading the run…