AgenticVBench-Omni

AgenticVBench-Omni / Transport and operations

Siren Yield

carla-siren-yield-ledger

You have a 90-minute recording of a single drive through the city of Varnholt, made by one car (called "the car" below) that carries four roof cameras and a four-microphone roof array, together with the chapter of the local traffic bylaw that covers emergency vehicles.

Family
Transport and operations
Inputs
videoaudiodocuments
Media
4 videos, 6 h, no sound track; 1 audio file, 1 h 30 min
Reference answer
31 vehicles
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Siren yield ledger

You have a 90-minute recording of a single drive through the city of Varnholt, made by one car (called "the car" below) that carries four roof cameras and a four-microphone roof array, together with the chapter of the local traffic bylaw that covers emergency vehicles.

Files (in the working directory)

  • cameras/front.mp4, cameras/right.mp4, cameras/rear.mp4, cameras/left.mp4 — silent video, 1280×720, 10 frames per second. All four start at the same instant: frame n of every file shows time t = n / 10 seconds. All times you report are on this video timeline.
  • audio/roof_mics.wav — 4-channel audio, 24 kHz. The audio recorder was started separately, somewhere between 0 and 60 seconds after the cameras, and ran until the cameras stopped; nothing in the files states the exact offset.
  • docs/traffic_code_chapter7.md — Chapter 7 of the Varnholt Road Traffic Bylaw.
Sensor rig
  • Cameras. All four sit at the centre of the car's roof, 2.0 m above the road, level with the horizon, each with a 90° horizontal field of view. front looks straight ahead, right looks 90° to the right of straight ahead, rear looks straight back, left looks 90° to the left. Parts of the car's own roof are visible at the image edges.
  • Microphones. Four cardioid microphones at the corners of the roof, 0.9 m apart, 1.6 m above the road. Channel 1 points 45° left of straight ahead (front-left), channel 2 45° right (front-right), channel 3 135° left (rear-left), channel 4 135° right (rear-right). A cardioid's gain for sound arriving at angle θ from the direction it points is 0.5·(1 + cos θ).
  • Heading. The city's streets run north–south and east–west. At t = 0 the car is heading north (to within 15°).
  • Sound. Sound travels in a straight line from its source to the microphones at 343 m/s and weakens with distance. A building standing between a source and the car muffles the sound but does not change the direction it arrives from. There are no echoes.

What to report

Emergency vehicles in Varnholt are police cars, ambulances and fire engines. During the drive some of them sound their sirens; others drive by without a siren, with or without flashing lights. Car alarms and reversing beepers also go off around the city; they are not sirens. Report every emergency vehicle whose siren sounds during the recording — one entry per vehicle. Each such vehicle sounds its siren for one continuous period, and every siren switches on abruptly (it does not fade in). Do not report emergency vehicles that never sound a siren.

For each vehicle give:

  • audible_s — the time (video timeline, seconds) at which its siren first becomes audible, i.e. the moment the sound of the siren switching on first reaches the car's microphones. Tolerance ±0.5 s.
  • direction — the compass direction the siren's sound arrives from at audible_s (the direction from the car towards where the sound comes from), as one of eight 45° sectors of compass bearing measured clockwise from north: N (337.5° to 22.5°), NE (22.5° to 67.5°), E (67.5° to 112.5°), SE (112.5° to 157.5°), S (157.5° to 202.5°), SW (202.5° to 247.5°), W (247.5° to 292.5°), NW (292.5° to 337.5°).
  • first_seen_s and first_seen_camera — when and in which camera the vehicle first comes into view. A vehicle is in view in a camera's frame when at least 3000 pixels of that vehicle are directly visible in that 1280×720 frame — not hidden behind buildings, trees, other vehicles or the car's own body, and not outside the image. It comes into view in a camera at the first frame of the first stretch of at least 10 consecutive frames (1 s) in which it is in view in that camera; first_seen_s is that frame's time and first_seen_camera that camera (front, right, rear or left). If it comes into view in two cameras less than 1 s apart, either camera (with its own time) is accepted. Tolerance ±1.0 s. If the vehicle never comes into view in any camera, give null for both.
  • vehicle_type — police_car, ambulance or fire_engine; null if the vehicle never comes into view.
  • verdict — whether the car yielded to this vehicle under Chapter 7 of the bylaw: yielded, failed_to_yield, or not_required (no duty to yield arose). The bylaw's "audible time" is this vehicle's audible_s.

Output

Write output/siren_ledger.json:

{
  "vehicles": [
    {"audible_s": 812.4, "direction": "SW", "first_seen_s": 826.1, "first_seen_camera": "rear",
     "vehicle_type": "ambulance", "verdict": "yielded"},
    {"audible_s": 1490.0, "direction": "N", "first_seen_s": null, "first_seen_camera": null,
     "vehicle_type": null, "verdict": "not_required"}
  ]
}

One object per vehicle, in any order. An entry counts only if all of its fields are right, so leave out a vehicle you cannot work out rather than guessing its fields.

Write this file early and keep rewriting it as you confirm more vehicles. The session has a time limit, and whatever is in the file when it ends is what counts.

Rules

  • Work only inside the working directory. Do not read, write or search outside it.
  • There is no internet access, and pip install will not work. Do not rely on memory of any simulator, map or dataset; every entry must come from what is in these files.
  • Available tools are what is installed: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. You can view still images you extract.

Media

Frame from front.mp4
front.mp4 · 1620 s
Frame from left.mp4
left.mp4 · 1620 s
Frame from rear.mp4
rear.mp4 · 1620 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 6 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, MIT.

Score

AgentScore (0 to 1)
GPT-6-Astra0.121
Claude Opus 5.50.708
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets a 90-minute simulated drive recorded by four roof cameras and a four-microphone array, plus a chapter of a traffic bylaw. For every emergency vehicle that sounds its siren it reports when the siren is first heard and from which compass direction, when and in which camera the vehicle first comes into view, what kind of vehicle it is, and whether the car yielded as the bylaw requires. The reference answer holds 31 vehicles.

What a good answer has to get right, and how the verifier checks it

Every siren vehicle is found, and none is invented
The score is an F1 over vehicles with one-to-one matching. A missed vehicle lowers recall; an entry that matches no vehicle, or cannot be read, lowers precision.
The siren is timed and its direction heard
The time the siren is first audible must be within 0.5 s, on the video’s timeline, and the direction must be the exact one of eight compass sectors. The microphone recording has its own clock, so its offset from the video has to be recovered first.
The vehicle is seen
The camera must be one in which the vehicle first comes into view, and the time within 1.0 s of that camera’s first-view time. In view means at least 3000 pixels of the vehicle visible for a second. Where two cameras pick it up less than a second apart, either is accepted with its own time.
The vehicle is identified
The type must be exact: police car, ambulance or fire engine.
The bylaw is applied
The verdict must be exact: yielded, failed to yield, or not required. It is computed from the simulator’s log under the rules of the bylaw chapter given to the agent.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 31 vehicles. Every vehicle counts the same, and an entry counts only when all of its fields are right: a vehicle heard at the right moment with the wrong verdict is a miss and a false positive.

What scores zero

No siren_ledger.json, a file that cannot be read or whose “vehicles” is not a list, an empty list, or a list in which no entry matches a vehicle on every field.

Outside the score: 124 further siren vehicles in the drive that the audit judged not observable from the recordings. They are excused: an unmatched entry whose siren time is within 1.5 s of one is left out of the count, so reporting one neither helps nor costs. The verifier also reports per-field agreement as diagnostics that do not enter the score.

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.12
Wall time
20.2 min
Steps shown
95

Loading the run…