The agent gets the recordings of fourteen meetings of a small city commission, with picture and sound in separate files, plus the agendas and a roster with seating charts. It lists every motion: when it was moved and when its result was announced, who moved and seconded it, the outcome, how each of the six commissioners voted, which camera view was live at those two moments, and the earlier motion it follows from, if any. The reference holds 103 motions.
What a good answer has to get right, and how the verifier checks it
- Every motion is found under the right item
- A motion is identified by its meeting, agenda item and order within that item. An entry whose key is not in the reference, or repeats one already used, is a false positive.
- The motion and its result are timed
- The moment it was moved and the moment the result was announced must each be within 2 s.
- Mover and seconder are named
- Both must be exact. They are checked only where the recording itself makes the name certain: the name is spoken with the role, or the voice matches clearly.
- The outcome and every vote are right
- The outcome must be exact, and so must the vote recorded for each of the six commissioners.
- The picture is used as well as the sound
- The camera view that was live when the motion was made, and when its result was announced, must both be exact.
- Linked motions are linked
- Where a motion amends or follows an earlier one, the link to that motion must be right; where it does not, none may be given.
How the checks become one score
Score = F1 = 2 × matched / (103 + entries counted). Every motion counts the same, and a motion counts only when every checked field is right: one wrong vote makes it a miss and a false positive.
What scores zero
No answer or one that cannot be read, a “motions” field that is not a list, an empty list, or a list in which no motion is right on every checked field.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Matched | Listed |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 103 | 103 |
| Every time 1 s offEach time is reported 1 s late or early, alternately. | 1.000 | 103 | 103 |
| One kind of mistake | |||
| Every motion recorded as unanimousEach “no” vote is recorded as “yes” (15 motions have one); absences are kept. Everything else is correct. | 0.854 | 88 | 103 |
| The first 7 meetings onlyMotions of the first 7 of 14 meetings are correct; the other meetings are left out. | 0.704 | 56 | 56 |
| Camera view always the daisThe live camera view is given as the dais at both moments of every motion. Everything else is correct. | 0.534 | 55 | 103 |
| Mover and seconder swappedEverything else is correct. Motions where neither name is checked still count. | 0.204 | 21 | 103 |
| Every time 5 s lateEverything else is correct. | 0.000 | 0 | 103 |
| No real work | |||
| Items and outcomes, nothing elseEvery motion is listed under the right item with its outcome, with all times at zero and no names, votes or views. | 0.000 | 0 | 103 |
| No file | 0.000 | 0 | 0 |
| An agent’s answer | |||
| A sampled runClaude Opus 5.5 · Claude Code 2.1.280. The run, step by step | 0.359 | 37 | 103 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: 13 further motions that the reference marks as don’t-care, for which entries are dropped, and any field the reference leaves empty because it could not be confirmed from the recordings. The verifier also reports accuracy per field, as diagnostics.