The agent gets a 10-minute silent montage of surveillance footage and pictures of ten people. For each pictured person it lists every occurrence of six activities, such as picking up an object or entering a vehicle, with start and end times. The reference holds 29 activity occurrences.
What a good answer has to get right, and how the verifier checks it
- The activity is given to the right person
- A listed occurrence can only be paired with a true occurrence of the same person and the same activity type.
- The occurrence is placed in time
- The midpoint of the listed interval must be within 3 s of the true midpoint; beyond that the pair scores 0.
- Closer timing earns more
- A pair scores 0.5 × (1 − midpoint error / 3 s) + 0.3 × interval overlap (intersection over union) + 0.2 × the ratio of the shorter duration to the longer. An exact interval scores 1.
- Occurrences are neither invented nor missed
- Within each person and activity, pairs are chosen in time order to give the largest total. The total is turned into two F1 values: one over all occurrences, 2 × total / (listed + 29), and one computed per person and averaged over the ten people.
How the checks become one score
Score = overall F1 × per-person mean F1. Multiplying the two means that doing well on the people with many occurrences cannot make up for getting other people wrong. A person with no true occurrence and none listed counts as 1 in the per-person mean.
What scores zero
No solution file or one over 1 MB, or a file with any structural error: an unknown or repeated person ID, an unknown activity type, a non-numeric time, or an interval that is empty or runs past the 600 s of the montage. One such entry voids the whole file. Otherwise, an answer in which no occurrence is within 3 s.
Outside the score: activities other than the six listed, and people who are not among the ten pictured.