AgenticVBench-Omni

AgenticVBench-Omni / Sports analysis

Chess

otb-chess-move-capture-reconstruction

You are given one silent video at /workspace/materials/game.mp4. It shows a real over-the-board chess game from the initial position through the end of the game. Some nonessential on-screen graphics have been masked out. Reconstruct the game record from the visible physical board.

Family
Sports analysis
Inputs
video
Media
1 video, 26 min, no sound track
Reference answer
104 moves and 26 captures
Deliverable
/workspace/output/solution.json
Scoring
checklist / accuracy

Task package · verifier code

Instruction

Full instruction given to the agent

Over-The-Board Chess Move Reconstruction

You are given one silent video at /workspace/materials/game.mp4. It shows a real over-the-board chess game from the initial position through the end of the game. Some nonessential on-screen graphics have been masked out. Reconstruct the game record from the visible physical board.

Write /workspace/output/solution.json in exactly this shape:

{
  "game_type": "chess",
  "result": "white_win | black_win | draw | unknown",
  "moves": [
    {
      "ply": 1,
      "move_number": 1,
      "side": "white | black",
      "san": "Nf3",
      "uci": "g1f3",
      "video_time": "HH:MM:SS.s"
    }
  ],
  "capture_events": [
    {
      "ply": 32,
      "move_number": 16,
      "side": "white | black",
      "san": "Nxg3",
      "uci": "e4g3",
      "capture_square": "g3",
      "captured_piece": "white_pawn | white_knight | white_bishop | white_rook | white_queen | white_king | black_pawn | black_knight | black_bishop | black_rook | black_queen | black_king",
      "video_time": "HH:MM:SS.s"
    }
  ]
}

Definitions:

  • ply: half-move number, starting at 1 for White's first move.
  • move_number: full chess move number.
  • side: the side making the move.
  • san: standard algebraic notation for the move.
  • uci: coordinate notation from origin square to destination square, with a promotion suffix if applicable, such as e7e8q.
  • video_time: the approximate time in the supplied video when the move has become visible on the physical board. Use HH:MM:SS.s. The verifier accepts timestamps within +/- 6 seconds of the reference annotation.
  • capture_events: include every move that captures a piece. Each capture event must repeat the move identity and report the square where the capture occurs and the color/type of the captured piece.

Rules:

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on memory of this game.
  • Use the video as evidence for every move and timestamp.
  • Do not include comments, prose, or extra top-level fields in the JSON.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.643
Claude Opus 5.50.640
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.014

One run per agent.

The agent watches a silent video of a real over-the-board chess game and writes the game record: every move with the time it was played, every capture with its square and the piece taken, and the result. The reference holds 104 moves and 26 captures.

What a good answer has to get right, and how the verifier checks it

The result is right
One check: the result of the game.
Every move is identified
For each of the 104 reference moves, an entry must give the same side and the same move, in either of two standard notations, at a position in the list no more than 3 moves away from the true one. One check per move.
Every move is timed
The matched entry’s video time must be within 6 s. One further check per move, which fails if the move was not identified.
Every capture is recorded
For each of the 26 captures: one check that it is identified, one that the capture square and the captured piece are right, and one that its time is within 6 s.
Nothing is added
Every extra move or capture in the answer adds one failed check.

How the checks become one score

Score = checks passed / checks made. A complete answer is judged on 287 checks: 1 for the result, 208 for moves and 78 for captures. This verifier counts checks instead of computing an F1, so a move with the right identity and the wrong time keeps half of its credit.

What scores zero

No solution file or one that cannot be read, or an answer with the wrong result and no move or capture identified.

Scores from this verifier: the reference answer with one thing changed

SubmissionScorePassedChecks
Careful work
The reference answer1.000287287
Every time 4 s offEach move and capture time is reported 4 s late or early, alternately.1.000287287
One kind of mistake
Captured piece always a pawnEvery capture names a pawn of the right colour as the piece taken. Everything else is correct.0.958275287
Moves onlyAll moves are correct; the list of captures is left empty.0.728209287
Every time 10 s lateAll moves and captures are identified; every time is 10 s late.0.547157287
The first 52 of 104 moves onlyMoves and captures of the first half of the game are correct; the rest and the result are missing.0.488140287
No real work
The result and nothing else0.0041287
No file0.0000287
An agent’s answer
A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step0.643200311

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: nothing in the answer is ignored; every entry that matches no reference move or capture counts as a failed check.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
TIME_TOLERANCE_S6constant in the verifier code
PLY_MATCH_WINDOW3constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.64
Wall time
7.4 min
Steps shown
63

Loading the run…