The agent gets a 30-minute first-person walk through a house, recorded with a microphone in each ear, and the floor plan. Hidden sources switch on and off around the house and none is visible. For each source the agent reports its room, its position on the plan and every interval during which it was on. The reference answer holds 24 sources with 98 on-intervals.
What a good answer has to get right, and how the verifier checks it
- Each source is placed in the right room
- A reported source can only be paired with a true source whose room label is identical, out of eight rooms.
- Each source is located on the plan
- Its position must be within 1.0 m of the true position, measured on the floor plan. Sources are paired one to one, in the way that matches the most intervals.
- Each switch-on and switch-off is timed
- An on-interval counts when its start and its end are each within 5 s of a true on-interval of the paired source.
- Intervals are neither invented nor missed
- The score is an F1 over on-intervals, not over sources, so a source that switches on five times weighs five times as much as one that is on once.
How the checks become one score
Score = F1 = 2 × matched intervals / (reported intervals + 98). An interval counts only when its source has the right room and position and both its ends are timed: a correctly timed interval attached to a source placed 2 m away counts as wrong.
What scores zero
No sources.json or one that cannot be read, no intervals at all, more than 100 sources or 2000 intervals, or an answer in which no interval matches.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Precision | Recall |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 1.000 | 1.000 |
| 0.5 m and 2 s offEvery source is placed 0.5 m to one side and every switch time is 2 s late. | 1.000 | 1.000 | 1.000 |
| One kind of mistake | |||
| Half of the sourcesEvery second source is left out; the others are correct. | 0.676 | 1.000 | 0.510 |
| Only the first interval of each sourceEach source is located correctly, and only its first on-interval is reported. | 0.393 | 1.000 | 0.245 |
| Right room, 1.5 m offEvery source is placed 1.5 m to one side, with the right room label. Times are correct. | 0.000 | 0.000 | 0.000 |
| Switch times 10 s lateRooms and positions are correct; every switch-on and switch-off is reported 10 s late. | 0.000 | 0.000 | 0.000 |
| One interval per sourceEach source is reported as on from its first switch-on to its last switch-off, with the gaps ignored. | 0.000 | 0.000 | 0.000 |
| No real work | |||
| One source per room, always onOne source at a fixed point in each of the eight rooms, on for the whole recording. | 0.000 | 0.000 | 0.000 |
| No file | 0.000 | 0.000 | 0.000 |
| An agent’s answer | |||
| A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step | 0.178 | 0.164 | 0.194 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the kind of sound each source makes, and the height of the source; only its position on the plan is compared.


