AgenticVBench-Omni

AgenticVBench-Omni / Transport and operations

Year of Driving

year-of-driving-route-ledger

One driver's car carries a forward-facing dashcam. You have 82 three-minute clips from drives made on 25 different days in one town, and for each day a street map of that town. From the video and the maps alone, reconstruct which streets the car drove in every clip and exactly when it got onto and off each of them, and when it passed a set of mapped places.

Family
Transport and operations
Inputs
videodocuments
Media
82 videos, 4 h 6 min, no sound track
Reference answer
205 street segments and 63 place passes
Deliverable
/workspace/output
Scoring
F1 / precision–recallJaccardper-component

Task package · verifier code

Instruction

Full instruction given to the agent

Same Town, Many Days: Route Ledger

One driver's car carries a forward-facing dashcam. You have 82 three-minute clips from drives made on 25 different days in one town, and for each day a street map of that town. From the video and the maps alone, reconstruct which streets the car drove in every clip and exactly when it got onto and off each of them, and when it passed a set of mapped places.

What is in the working directory

  • days/dayNN/cMM.mp4: the clips, grouped by day. Every clip is exactly 180 seconds long, 1920x908, 15 fps, no audio. Day numbers are arbitrary labels, not calendar order, and within a day the clip numbers are in random order. Clips of the same day never overlap; one clip may continue straight on from another, possibly after a gap. The car sometimes starts from, or stops at, the same spot on different days. The camera's date, time, speed and GPS text was cropped away; licence plates, faces and roadside signs (street-name signs, route shields, shop and pole signs, billboards) are blurred; the files carry no timestamps or other metadata.
  • days/dayNN/map/: that day's street map.
  • overview.png: the whole mapped area, a disc about 16 km across, with the grid of map sheets drawn in blue.
  • sheets/sheet_*.png: zoomed map sheets (1.5 m per pixel, overlapping edges). Listed streets are drawn darker and labelled with their street IDs; roads without a street ID (ramps, service roads, parking aisles, unnamed roads) are drawn lighter. Places are red dots labelled with place IDs. Axis ticks give map coordinates in metres. The maps show roads only: no names, road classes, railways or rivers.
  • places.csv: place_id, street_id for every place; street_id is the street the red dot sits on.

Every day's map covers the same area and the same roads, but each is drawn at its own rotation (up on the sheet is not north, and the rotation differs from day to day; scale is true), and each day uses its own labels: street IDs and place IDs are shuffled from day to day. An ID refers to a street or place only on that day's map. Always answer a clip with the labels of its own day's map.

Every clip stays inside the mapped area. All times are seconds from the start of the clip.

Part 1: route segments

For each clip, list the listed streets the car travels along, in driving order.

  • A street gets an entry when the car travels along it for at least 40 m in one go, measured along the road on the map. Streets that are only crossed, or driven for less than 40 m, get no entry. Stretches of roughly 20 to 40 m are accepted whether you list them or not; if you leave one out, merge what it separated as below.
  • Roads without a street ID are ignored: they get no entry and do not split an entry.
  • Consecutive entries are always different streets. After leaving out what does not count, merge repeats: S0010, then a parking lot, then S0010 again is a single S0010 entry, from the first join to the last exit. A street driven again after a different listed street gets a new entry.
  • join_s is the moment the car passes from the previous road onto this street: for a turn, the middle of the turn; where one street continues straight into another, the moment it crosses the point on the map where the street ID changes; after a road without a street ID, the moment it gets onto this street. A street the clip begins on has join_s 0.
  • exit_s is the moment the car leaves the street, defined the same way (normally the next entry's join_s, or the moment it turns off onto a road without a street ID). A street the clip ends on has exit_s 180.
  • An entry counts only if its street is right and both join_s and exit_s are within 15 seconds of the true moments. Entries that match nothing count against you.

Part 2: place passes

For each clip, list every time the car passes a place of that day's map. The car passes a place when it travels along the place's street past the red dot, within 40 m of it. Being on a different street nearby (a parallel road or a cross street) does not count, and neither does turning off the place's street before reaching the dot. Going past the same dot twice gives two passes. Some places are not passed in any clip.

  • t_s is the moment the car is closest to the dot. A pass counts only if place and time (within 15 seconds) are right.

What to submit

Write /workspace/output/route_ledger.json in exactly this shape:

{
  "clips": {
    "day01/c01": {
      "route": [
        {"street_id": "S0123", "join_s": 0, "exit_s": 37.5},
        {"street_id": "S0456", "join_s": 37.5, "exit_s": 180}
      ],
      "passes": [
        {"place_id": "P07", "t_s": 101}
      ]
    }
  }
}
  • clips has one key per clip, dayNN/cMM (the file path without days/ and .mp4).
  • Write this file early and keep overwriting it. As soon as you have one clip, write the file with what you have, and rewrite it each time you finish another. There is a time limit; whatever is in the file when the session ends is what gets graded.

Rules

  • Work only inside the working directory (/workspace). Do not read, write, or search outside it.
  • There is no internet access and none is needed. Do not rely on memory or outside knowledge of this town, these roads, or where the recordings came from; every entry must be justified by what you see in the clips and the supplied maps.
  • Tools available are what is installed: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless and matplotlib. pip install will not work.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 732 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, GPL-3.0, ODbL, odbl.

Score

AgentScore (0 to 1)
GPT-6-Astra0.057
Claude Opus 5.50.006
Claude Opus 50.002
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets 82 three-minute dashcam clips from drives on 25 different days in one town, with a street map for each day and all signs and on-screen text removed. For every clip it lists the streets the car drove, with the moment it got onto and off each, and every time it passed a mapped place. The reference, from the logged drives, holds 205 street segments and 63 place passes.

What a good answer has to get right, and how the verifier checks it

Each street is identified
A route entry can only count for a logged segment of the same clip with the same street ID from that day’s map.
Getting onto and off the street is timed
The join time and the exit time must each be within 15 s of the logged moments, and a logged segment can be matched once. A street the clip begins or ends on has the times 0 and 180.
Each place pass is found
A pass counts when the place is right and the time of closest approach is within 15 s. Driving a parallel or crossing street near the place is not a pass.
Nothing is listed that did not happen
Every route entry and every pass that matches nothing is a false positive, and every logged item that is not matched is a miss.

How the checks become one score

Score = hits / (hits + false positives + misses), counted over all route segments and place passes of all 82 clips together. This is a Jaccard index, which is stricter than F1: an answer with half the items right and nothing wrong scores 0.5, where F1 would give 0.67. Segments and passes are pooled, so the 205 segments weigh more than the 63 passes.

What scores zero

No route_ledger.json or one that cannot be read, or an answer in which no segment and no pass matches.

Outside the score: nothing in the answer is ignored. The verifier also reports route and pass scores separately, as diagnostics.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.06
Wall time
58.9 min
Steps shown
163

Loading the run…