The agent watches a silent video of a real over-the-board chess game and writes the game record: every move with the time it was played, every capture with its square and the piece taken, and the result. The reference holds 104 moves and 26 captures.
What a good answer has to get right, and how the verifier checks it
- The result is right
- One check: the result of the game.
- Every move is identified
- For each of the 104 reference moves, an entry must give the same side and the same move, in either of two standard notations, at a position in the list no more than 3 moves away from the true one. One check per move.
- Every move is timed
- The matched entry’s video time must be within 6 s. One further check per move, which fails if the move was not identified.
- Every capture is recorded
- For each of the 26 captures: one check that it is identified, one that the capture square and the captured piece are right, and one that its time is within 6 s.
- Nothing is added
- Every extra move or capture in the answer adds one failed check.
How the checks become one score
Score = checks passed / checks made. A complete answer is judged on 287 checks: 1 for the result, 208 for moves and 78 for captures. This verifier counts checks instead of computing an F1, so a move with the right identity and the wrong time keeps half of its credit.
What scores zero
No solution file or one that cannot be read, or an answer with the wrong result and no move or capture identified.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Passed | Checks |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 287 | 287 |
| Every time 4 s offEach move and capture time is reported 4 s late or early, alternately. | 1.000 | 287 | 287 |
| One kind of mistake | |||
| Captured piece always a pawnEvery capture names a pawn of the right colour as the piece taken. Everything else is correct. | 0.958 | 275 | 287 |
| Moves onlyAll moves are correct; the list of captures is left empty. | 0.728 | 209 | 287 |
| Every time 10 s lateAll moves and captures are identified; every time is 10 s late. | 0.547 | 157 | 287 |
| The first 52 of 104 moves onlyMoves and captures of the first half of the game are correct; the rest and the result are missing. | 0.488 | 140 | 287 |
| No real work | |||
| The result and nothing else | 0.004 | 1 | 287 |
| No file | 0.000 | 0 | 287 |
| An agent’s answer | |||
| A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step | 0.643 | 200 | 311 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: nothing in the answer is ignored; every entry that matches no reference move or capture counts as a failed check.