The agent watches a silent first-person Doom video of six episodes and lists every scored interaction in order (key and weapon pickups, switches, locked doors opening, checkpoints activated and restored, the exit), each with its time and the complete state right after it: active weapon, keys held, switches on, doors open, current checkpoint. The reference holds 23 events per episode, 138 in all.
What a good answer has to get right, and how the verifier checks it
- Every interaction is found, in order
- Within each episode, entries are aligned with the reference in order. A pair can align only when the event type and the object are the same and the time is within 750 ms.
- The full state after each event is right
- An aligned event counts in full only when all five state fields are exactly right. This gives the full-event F1 = 2 × full matches / (reference events + reported events).
- State is carried forward correctly
- State depends on everything before it: keys are kept, doors stay open, and restoring a checkpoint puts the switches back to how they were when it was activated. One early mistake therefore shows up in every later state.
- Partly right states earn a little
- For aligned events, each of the five state fields is also scored on its own with the same F1 formula, and the five are averaged.
How the checks become one score
Each episode’s score = 0.9 × full-event F1 + 0.1 × mean per-field F1. Score = the mean over the six episodes. The 0.1 part is the only credit for an event found at the right time with a state that is not completely right.
What scores zero
No solution file, one over 1 MB, or one whose top level is not exactly an “episodes” list. An episode that is missing, listed twice, carries extra keys or has more than 100 events scores 0 for that episode. Otherwise, an answer in which no event aligns.
Outside the score: health, ammo and armor pickups, objects looked at but not used, and failed uses of doors and switches, which the task names as distractors. An entry that breaks the event format is dropped before alignment.