The agent watches the full broadcast of a men’s lacrosse game, silent and with the scoreboard blacked out, and lists every goal in order: the team, the scorer’s jersey number, and the assister’s number or none. The reference holds 26 goals.
What a good answer has to get right, and how the verifier checks it
- Every goal is found, in order
- The answer is compared with the reference as a sequence: the verifier counts the longest run of goals that appear in both lists in the same order.
- The team and the scorer are right
- The team, read from jersey colour, and the scorer’s number must both be exact. Numbers repeat across the two teams, so the number alone is not enough.
- Assisted or not
- The entry must say correctly whether the goal was assisted. The assister’s number itself is not required: an audit found it unreadable in the video for about half of the assisted goals.
- The running score is right
- Each goal also carries the score after it, computed from the answer’s own sequence of teams. A missed or invented goal therefore makes every later goal of that stretch wrong until the count is back in step.
How the checks become one score
Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 26 goals and precision over the goals listed. A goal counts only when team, scorer, assisted-or-not and running score are all right.
What scores zero
No answer or one that cannot be read, an empty list, or a list in which no goal is right on all four items.
Outside the score: the assister’s number and the penalties, which the verifier reports as diagnostics without adding them to the score.