AgenticVBench-Omni

AgenticVBench-Omni / Transport and operations

Driving Infractions

carla-surround-infraction-ledger

You are given two continuous hours of driving by a single car through a small town, recorded by eight cameras mounted together on the car's roof, plus a map of the town. The car is driven badly on purpose: sometimes it enters junctions against a red light, fails to stop properly at STOP approaches, or drives too fast. It also often does the lawful thing at the last moment. Your job is to list every traffic infraction it commits.

Family
Transport and operations
Inputs
videodocuments
Media
8 videos, 16 h, no sound track
Reference answer
136 infractions
Deliverable
/workspace/output
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

Surround-Camera Infraction Ledger

You are given two continuous hours of driving by a single car through a small town, recorded by eight cameras mounted together on the car's roof, plus a map of the town. The car is driven badly on purpose: sometimes it enters junctions against a red light, fails to stop properly at STOP approaches, or drives too fast. It also often does the lawful thing at the last moment. Your job is to list every traffic infraction it commits.

Files (in the working directory)

  • cameras/front.mp4, cameras/front_right.mp4, cameras/right.mp4, cameras/rear_right.mp4, cameras/rear.mp4, cameras/rear_left.mp4, cameras/left.mp4, cameras/front_left.mp4 - silent 1280x720 video at 10 frames per second, 72,000 frames each. All eight are synchronised: frame k of every file was captured at the same instant, t = k / 10 seconds after the start. There is no speedometer, clock or other overlay.
  • Camera rig: the eight cameras share one mounting point on the roof and identical lenses, and are mounted level. They point 0, 45, 90, 135, 180, 225, 270 and 315 degrees clockwise from straight ahead (front, front_right, right, rear_right, rear, rear_left, left, front_left). The car's front bumper is 2.15 m ahead of the mounting point. The rig is not perfectly rigid and shakes a little as the car drives.
  • map/town_map.png - top-down map, north up, with metre axes and a 100 m scale bar. It shows every road with white lane direction arrows (traffic drives on the right), every junction as a dashed outline labelled J<number>, every traffic-light approach as a green diamond labelled L<number> drawn at that approach's stop line, every STOP-controlled approach as a red octagon labelled S<number>, every speed-limit sign as a blue triangle labelled P<number> (the number on the sign is not shown on the map), and the car's starting position and heading (orange START arrow).
  • map/town_map.json - the labelled features of the map in metres (x grows east, y grows south; headings in degrees with 0 = east and 90 = south): junction outlines, the stop-line position and approach heading of every L approach, the position and approach heading of every S approach, P sign positions, and the start pose. Road geometry is only in the image.

The town has no other moving traffic and no pedestrians; parked cars are scenery.

Checkpoints

Infractions are judged only at two kinds of moment, called checkpoints:

  • Junction entry - the moment the car's front bumper crosses into a junction outline on the map.
  • Sign pass - the moment the car's front bumper draws level with a speed-limit sign that applies to the car's direction of travel (its face is toward the approaching car). Signs for the other carriageway do not count.

Infraction types (closed vocabulary)

  • red_light - at a junction entry from a traffic-light approach, that approach's light shows red. Entering on green or on yellow is not an infraction, and neither is starting to move early if the light has turned green by the entry moment. location = the junction, rule = the L id of the approach the car came from.
  • stop_sign - at a junction entry from a STOP approach, the car has not come to a complete stop since it left the previous junction. A complete stop means the car is fully stationary for at least 1.0 second continuously; shorter pauses and slow rolling do not count. location = the junction, rule = the S id of the approach the car came from.
  • speeding - at a checkpoint (either kind), the car's speed is above the speed limit in force. The limit in force is the number, in km/h, on the most recent speed-limit sign the car passed before that checkpoint; it stays in force through junctions and turns until the car passes another speed-limit sign. At a sign pass, the sign being passed does not yet count - the limit is still the previous one. Before the car has passed its first sign, no limit is in force. location = the junction (J..) for a junction entry or the sign (P..) for a sign pass, rule = the limit in force, e.g. "30".

One checkpoint can produce more than one infraction (for example red_light and speeding at the same junction entry); list each one separately. Most checkpoints are lawful.

Output

Write output/infractions.json:

{
  "infractions": [
    {"type": "red_light", "time_s": 1234.5, "location": "J1427", "rule": "L28"},
    {"type": "stop_sign", "time_s": 2201.0, "location": "J562", "rule": "S08"},
    {"type": "speeding", "time_s": 951.7, "location": "J838", "rule": "30"},
    {"type": "speeding", "time_s": 5874.2, "location": "P01", "rule": "90"}
  ]
}
  • type: one of red_light, stop_sign, speeding.
  • time_s: seconds from the start of the videos (frame index / 10) at the checkpoint moment. An item counts only if it is within 0.6 seconds of the true checkpoint moment.
  • location and rule: ids exactly as printed on the map (and the limit as a number string for speeding). An item counts only if type, location and rule are all correct.
  • Order does not matter. Do not list lawful checkpoints.

Write the file early and overwrite it as you go: the session has a time limit, and whatever is in output/infractions.json when it ends is what is graded.

Rules

  • Work only inside the working directory. There is no internet access. Do not rely on memory of any simulator, town or dataset; every item must come from what you observe in these files.
  • Tools available: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless and matplotlib. Nothing else can be installed.

Media

Frame from front.mp4
front.mp4 · 2160 s
Frame from front_left.mp4
front_left.mp4 · 2160 s
Frame from front_right.mp4
front_right.mp4 · 2160 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 10 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, MIT.

Score

AgentScore (0 to 1)
GPT-6-Astra0.249
Claude Opus 5.50.641
Claude Opus 50.025
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches two hours of driving recorded by eight cameras and lists every traffic infraction with its time, its place and the rule that was broken. The reference answer holds 136 infractions among 503 checkpoints.

What a good answer has to get right, and how the verifier checks it

Every infraction is found, and nothing lawful is listed
The score is an F1 over the 136 infractions with one-to-one matching. A missed infraction lowers recall; a lawful checkpoint listed as an infraction lowers precision.
The rule that was broken is named
An item counts only when its type, its junction or sign, and its rule all match exactly: the light or STOP approach the car came from, or the speed limit in force at that moment.
The written rules are applied as written
The reference is computed from the simulator’s log under the rules in the instruction: the light is red at the moment the front bumper enters the junction (yellow is lawful); the car has not stood still for a full second before a STOP junction (a shorter pause is not a stop); the speed is above the limit on the last sign passed.
It is placed at the right moment
The time must be within 0.6 s of the checkpoint. An observer model with errors of ±0.3 s in time, ±4% in speed and ±1 frame in stillness scores 1.0.
Borderline cases are not held against the answer
Cases too close to call are excused: red within 0.3 s of a phase change, a stop of 0.8 to 0.9 s, a speed within 2 km/h of the limit. Listing one neither helps nor costs. This drive has 1.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall) over the 136 infractions. Every infraction counts the same, and an item counts only when its type, place, rule and time are all right.

What scores zero

No answer file or one that cannot be read, an empty list, or a list in which no item matches an infraction on every field.

Scores from this verifier: the reference answer with one thing changed

SubmissionScorePrecisionRecall
Careful work
The reference answer1.0001.0001.000
Every time 0.3 s offEach infraction's time is reported 0.3 s late or 0.3 s early, alternately.1.0001.0001.000
One kind of mistake
Any pause counts as a stopThe 17 of 30 stop-sign infractions where the car paused for under a second are left out. Everything else is correct.0.9331.0000.875
Yellow counted as redThe 72 junction entries on a yellow light are added as red-light infractions. Everything else is correct.0.7910.6541.000
Speed limit never updatedSpeeding is judged against the first sign's limit (30 km/h) for the whole drive. Red-light and stop-sign items are correct.0.6210.4960.831
Every checkpoint flaggedEvery entry from a light or STOP approach is reported as an infraction, and speeding at every checkpoint, each with the exact time, place and rule id.0.2510.1431.000
No real work
Right counts, guessed itemsThe packaged guess with the right number of items of each type at made-up times and places.0.0000.0000.000
No file0.0000.0000.000
An agent’s answer
A sampled runClaude Opus 5.5 · Claude Code 2.1.280. The run, step by step0.6410.6670.618

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the 1 excused case, and anything the task does not ask for, such as an explanation of each infraction.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
details.time_tol_s0.6value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.64
Wall time
164 min
Steps shown
232

Loading the run…