The agent watches a 25-minute film of two player-less pianos performing four pieces, reads the printed scores, and lists every note where the performance departs from the score: piece, bar, beat, piano, type of departure and pitch. The reference answer holds 186 departures.
What a good answer has to get right, and how the verifier checks it
- Every departure is found, and none is invented
- The score is an F1 over departures with one-to-one matching. A missed departure lowers recall. Every entry in the file counts as a claim, including one the verifier cannot read, so an entry that matches nothing lowers precision.
- The note is located in the printed score
- Piece and bar must be exact, and the beat must be within 0.125 of a quarter note.
- The right piano is named
- The piano, I or II, must be the one that departed from its own part. Both pianos are mixed into one mono track with the same sound, so this has to be read from the picture.
- The kind of departure is right
- The type must be exactly one of four: wrong note, octave error, omitted note or extra note.
- The pitch is right
- Pitches are compared as MIDI note numbers, so C#4 and Db4 are the same answer. For an omitted note it is the printed pitch; otherwise the pitch that was played.
How the checks become one score
Score = F1 = 2 × matched / (entries listed + 186). Every departure counts the same, and an entry counts only when all six fields are right.
What scores zero
No deviations.json, a file that is not valid JSON or whose “deviations” is not a list, an empty list, or a list in which no entry matches a departure on all six fields.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Precision | Recall |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 1.000 | 1.000 |
| Every beat 0.1 offEach beat is reported 0.1 of a quarter note late or early, alternately. | 1.000 | 1.000 | 1.000 |
| One kind of mistake | |||
| Omitted notes never noticedThe 14 departures of type omitted_note are left out. Everything else is correct. | 0.961 | 1.000 | 0.925 |
| Octave errors called wrong notesThe 33 octave errors are reported as wrong notes. Everything else is correct. | 0.823 | 0.823 | 0.823 |
| Only the first two piecesDepartures in pieces A and B only. | 0.602 | 1.000 | 0.430 |
| Bars counted from 1 instead of 0Every bar number is one too high, as when an incomplete first bar is counted as bar 1. | 0.027 | 0.027 | 0.027 |
| Right notes, wrong pianoEvery departure is attributed to the other piano. Everything else is correct. | 0.000 | 0.000 | 0.000 |
| No real work | |||
| Empty list | 0.000 | 0.000 | 0.000 |
| No file | 0.000 | 0.000 | 0.000 |
| An agent’s answer | |||
| A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step | 0.011 | 0.333 | 0.005 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: tempo, dynamics, articulation, pedalling and the usual way of playing printed ornaments, which the task says are never departures. The verifier also reports how many entries would match if the piano, the pitch, the type or the beat were ignored; those figures are diagnostics and do not enter the score.


