The agent watches 14 drone clips and writes, for each of 28 marked animals, a timeline of its behaviour and the seconds spent in each behaviour. The reference answer holds 274 bouts.
What a good answer has to get right, and how the verifier checks it
- Each marked animal is followed for the whole clip
- Each of the 28 animals is scored against its own reference timeline, and the scores are averaged. An animal with no entry scores 0.
- The behaviour is named correctly
- A reported bout counts only when its behaviour is the same as the reference bout’s, out of seven listed behaviours.
- Each change of behaviour is timed
- The bout’s start and end must both be within 1.5 s. The tolerance was set from a blind re-coding of 20 reference boundaries (19 could be coded), where the largest error was 1.44 s.
- Bouts are neither invented nor missed
- The timeline score is an F1 over bouts, so an extra bout and a missed bout both cost. A change lasting 2 to 4 s may be reported or folded into the bouts around it: the answer is scored at floors of 2, 3 and 4 s and the best is kept.
- Nothing is claimed about what cannot be seen
- Time the reference marks as not visible or as some other behaviour is not checked, and a boundary next to such time may fall anywhere inside it.
- The time budget adds up
- For each behaviour the total must be within 3 s or 10% of the reference, with more room for behaviours that have many boundaries. More than 3 s reported for a behaviour the animal never shows counts as wrong. The budget is half of each animal’s score.
How the checks become one score
Each animal’s score = 0.5 × timeline F1 + 0.5 × budget score. The budget score is the share of that animal’s behaviour totals that are within tolerance. Score = the mean over all 28 marked animals, each weighing the same whether its clip lasts two minutes or five.
What scores zero
No behaviour_budgets.json or one that cannot be read. An animal that is missing from the answer scores 0 and still counts in the mean, as does an animal with no bout matched and no budget value within tolerance.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Timeline | Budget |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 1.000 | 1.000 |
| Every boundary 0.5 s offEach change of behaviour is reported 0.5 s late or 0.5 s early, alternately, and the time budgets are added up from that shifted timeline. | 0.996 | 0.992 | 1.000 |
| One kind of mistake | |||
| Short bouts left outOnly bouts of 10 s or more are reported; shorter ones are absorbed into the bouts around them. | 0.519 | 0.362 | 0.677 |
| One look every 10 sThe animal is checked once every 10 s and that behaviour is held until the next check. | 0.423 | 0.140 | 0.705 |
| Never reports head_upEvery head_up bout is reported as graze. Everything else is correct. | 0.564 | 0.600 | 0.527 |
| Right timelines, wrong animalsIn clips with several marked animals, each animal is given the correct answer of another marked animal (22 of 28 animals are in such clips). | 0.277 | 0.229 | 0.324 |
| No real work | |||
| One behaviour per animalThe most common behaviour of each animal, held for the whole clip. | 0.069 | 0.000 | 0.138 |
| No file | 0.000 | 0.000 | 0.000 |
| An agent’s answer | |||
| A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step | 0.203 | 0.118 | 0.287 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: whatever is reported for time the reference marks as not visible or as some other behaviour, and anything the task does not ask for, such as a written interpretation.


