The agent watches two hours of driving recorded by eight cameras and lists every traffic infraction with its time, its place and the rule that was broken. The reference answer holds 136 infractions among 503 checkpoints.
What a good answer has to get right, and how the verifier checks it
- Every infraction is found, and nothing lawful is listed
- The score is an F1 over the 136 infractions with one-to-one matching. A missed infraction lowers recall; a lawful checkpoint listed as an infraction lowers precision.
- The rule that was broken is named
- An item counts only when its type, its junction or sign, and its rule all match exactly: the light or STOP approach the car came from, or the speed limit in force at that moment.
- The written rules are applied as written
- The reference is computed from the simulator’s log under the rules in the instruction: the light is red at the moment the front bumper enters the junction (yellow is lawful); the car has not stood still for a full second before a STOP junction (a shorter pause is not a stop); the speed is above the limit on the last sign passed.
- It is placed at the right moment
- The time must be within 0.6 s of the checkpoint. An observer model with errors of ±0.3 s in time, ±4% in speed and ±1 frame in stillness scores 1.0.
- Borderline cases are not held against the answer
- Cases too close to call are excused: red within 0.3 s of a phase change, a stop of 0.8 to 0.9 s, a speed within 2 km/h of the limit. Listing one neither helps nor costs. This drive has 1.
How the checks become one score
Score = F1 = 2 × precision × recall / (precision + recall) over the 136 infractions. Every infraction counts the same, and an item counts only when its type, place, rule and time are all right.
What scores zero
No answer file or one that cannot be read, an empty list, or a list in which no item matches an infraction on every field.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Precision | Recall |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 1.000 | 1.000 |
| Every time 0.3 s offEach infraction's time is reported 0.3 s late or 0.3 s early, alternately. | 1.000 | 1.000 | 1.000 |
| One kind of mistake | |||
| Any pause counts as a stopThe 17 of 30 stop-sign infractions where the car paused for under a second are left out. Everything else is correct. | 0.933 | 1.000 | 0.875 |
| Yellow counted as redThe 72 junction entries on a yellow light are added as red-light infractions. Everything else is correct. | 0.791 | 0.654 | 1.000 |
| Speed limit never updatedSpeeding is judged against the first sign's limit (30 km/h) for the whole drive. Red-light and stop-sign items are correct. | 0.621 | 0.496 | 0.831 |
| Every checkpoint flaggedEvery entry from a light or STOP approach is reported as an infraction, and speeding at every checkpoint, each with the exact time, place and rule id. | 0.251 | 0.143 | 1.000 |
| No real work | |||
| Right counts, guessed itemsThe packaged guess with the right number of items of each type at made-up times and places. | 0.000 | 0.000 | 0.000 |
| No file | 0.000 | 0.000 | 0.000 |
| An agent’s answer | |||
| A sampled runClaude Opus 5.5 · Claude Code 2.1.280. The run, step by step | 0.641 | 0.667 | 0.618 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the 1 excused case, and anything the task does not ask for, such as an explanation of each infraction.


