The agent gets the minimap view of one full League of Legends game, about 39.5 minutes, with no clock, kill feed, names or gold shown. It lists every key event: its time, its type (champion kill, tower kill, epic monster kill), who or what it concerns, and two judgements about gold at that moment. The reference holds 111 events.
What a good answer has to get right, and how the verifier checks it
- Every key event is found, and none is invented
- The score is an F1 over events; each reference event can be matched once, by the closest entry in time. A missed event lowers recall and an entry that matches nothing lowers precision.
- The event is timed
- The time must be within 3 s of the true game clock, read from the video’s own frame time.
- The event is identified
- Type and entity must both be exact: which champion died, which of the towers fell, which dragon or baron was taken. Champions appear only as small portraits on the map.
- The state of the economy is judged
- Two further fields must be exact: which side gained more gold during that minute (by 300 or more), and which side was ahead before it (by 1,000 or more), each blue, red or equal. Gold is never shown, so both have to be inferred from what happens on the map.
- A player can make that judgement from the map
- In a check made during development, two players ranked Emerald were given only the video and instruction that the agent gets. Each got about 85% of each gold field right, and their answers averaged 0.79 under this verifier. Their misses were mostly early in the game, where the difference between the sides is close to the thresholds.
How the checks become one score
Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 111 events. Every event counts the same, and an event counts only when time, type, entity and both gold fields are right.
What scores zero
No solution file or one that cannot be read, an empty list, or a list in which no entry is right on every field.
Scores from this verifier: the reference answer with one thing changed
| Submission | Score | Precision | Recall |
|---|---|---|---|
| Careful work | |||
| The reference answer | 1.000 | 1.000 | 1.000 |
| Every time 2 s offEach time is reported 2 s late or early, alternately. | 1.000 | 1.000 | 1.000 |
| One kind of mistake | |||
| Champion kills onlyThe 25 tower and epic-monster events are left out. Everything else is correct. | 0.873 | 1.000 | 0.775 |
| Kills found, champion wrongEvery champion kill names the champion of the next kill instead. Towers and monsters are correct. | 0.396 | 0.396 | 0.396 |
| Gold leader always blueThe side in the lead is given as blue for every event. Everything else is correct. | 0.360 | 0.360 | 0.360 |
| Gold never judgedBoth gold fields are given as “equal” for every event. Times, types and entities are correct. | 0.054 | 0.054 | 0.054 |
| Every time 5 s lateEverything else is correct. | 0.009 | 0.009 | 0.009 |
| No real work | |||
| Empty list | 0.000 | 0.000 | 0.000 |
| No file | 0.000 | 0.000 | 0.000 |
| An agent’s answer | |||
| A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step | 0.279 | 0.279 | 0.279 |
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: nothing in the answer is ignored. Answers in an earlier field layout are converted before scoring.