You have a 90-minute recording of a single drive through the city of Varnholt, made by one car (called "the car" below) that carries four roof cameras and a four-microphone roof array, together with the chapter of the local traffic bylaw that covers emergency vehicles.
Full instruction given to the agent
Siren yield ledger
You have a 90-minute recording of a single drive through the city of Varnholt, made by one car (called
"the car" below) that carries four roof cameras and a four-microphone roof array, together with the
chapter of the local traffic bylaw that covers emergency vehicles.
Files (in the working directory)
cameras/front.mp4, cameras/right.mp4, cameras/rear.mp4, cameras/left.mp4 — silent video,
1280×720, 10 frames per second. All four start at the same instant: frame n of every file shows
time t = n / 10 seconds. All times you report are on this video timeline.
audio/roof_mics.wav — 4-channel audio, 24 kHz. The audio recorder was started separately, somewhere
between 0 and 60 seconds after the cameras, and ran until the cameras stopped; nothing in the files
states the exact offset.
docs/traffic_code_chapter7.md — Chapter 7 of the Varnholt Road Traffic Bylaw.
Sensor rig
- Cameras. All four sit at the centre of the car's roof, 2.0 m above the road, level with the
horizon, each with a 90° horizontal field of view.
front looks straight ahead, right looks 90°
to the right of straight ahead, rear looks straight back, left looks 90° to the left. Parts of
the car's own roof are visible at the image edges.
- Microphones. Four cardioid microphones at the corners of the roof, 0.9 m apart, 1.6 m above the
road. Channel 1 points 45° left of straight ahead (front-left), channel 2 45° right (front-right),
channel 3 135° left (rear-left), channel 4 135° right (rear-right). A cardioid's gain for sound
arriving at angle θ from the direction it points is 0.5·(1 + cos θ).
- Heading. The city's streets run north–south and east–west. At
t = 0 the car is heading north
(to within 15°).
- Sound. Sound travels in a straight line from its source to the microphones at 343 m/s and
weakens with distance. A building standing between a source and the car muffles the sound but does
not change the direction it arrives from. There are no echoes.
What to report
Emergency vehicles in Varnholt are police cars, ambulances and fire engines. During the drive some of
them sound their sirens; others drive by without a siren, with or without flashing lights. Car alarms and
reversing beepers also go off around the city; they are not sirens. Report
every emergency vehicle whose siren sounds during the recording — one entry per vehicle. Each such
vehicle sounds its siren for one continuous period, and every siren switches on abruptly (it does not
fade in). Do not report emergency vehicles that never sound a siren.
For each vehicle give:
audible_s — the time (video timeline, seconds) at which its siren first becomes audible, i.e. the
moment the sound of the siren switching on first reaches the car's microphones. Tolerance ±0.5 s.
direction — the compass direction the siren's sound arrives from at audible_s (the direction from
the car towards where the sound comes from), as one of eight 45° sectors of compass bearing measured
clockwise from north: N (337.5° to 22.5°), NE (22.5° to 67.5°), E (67.5° to 112.5°), SE
(112.5° to 157.5°), S (157.5° to 202.5°), SW (202.5° to 247.5°), W (247.5° to 292.5°), NW
(292.5° to 337.5°).
first_seen_s and first_seen_camera — when and in which camera the vehicle first comes into
view. A vehicle is in view in a camera's frame when at least 3000 pixels of that vehicle are
directly visible in that 1280×720 frame — not hidden behind buildings, trees, other vehicles or the
car's own body, and not outside the image. It comes into view in a camera at the first frame of
the first stretch of at least 10 consecutive frames (1 s) in which it is in view in that camera;
first_seen_s is that frame's time and first_seen_camera that camera (front, right, rear
or left). If it comes into view in two cameras less than 1 s apart, either camera (with its own
time) is accepted. Tolerance ±1.0 s. If the vehicle never comes into view in any camera, give
null for both.
vehicle_type — police_car, ambulance or fire_engine; null if the vehicle never comes into
view.
verdict — whether the car yielded to this vehicle under Chapter 7 of the bylaw: yielded,
failed_to_yield, or not_required (no duty to yield arose). The bylaw's "audible time" is this
vehicle's audible_s.
Output
Write output/siren_ledger.json:
{
"vehicles": [
{"audible_s": 812.4, "direction": "SW", "first_seen_s": 826.1, "first_seen_camera": "rear",
"vehicle_type": "ambulance", "verdict": "yielded"},
{"audible_s": 1490.0, "direction": "N", "first_seen_s": null, "first_seen_camera": null,
"vehicle_type": null, "verdict": "not_required"}
]
}
One object per vehicle, in any order. An entry counts only if all of its fields are right, so leave out
a vehicle you cannot work out rather than guessing its fields.
Write this file early and keep rewriting it as you confirm more vehicles. The session has a time
limit, and whatever is in the file when it ends is what counts.
Rules
- Work only inside the working directory. Do not read, write or search outside it.
- There is no internet access, and
pip install will not work. Do not rely on memory of any
simulator, map or dataset; every entry must come from what is in these files.
- Available tools are what is installed:
ffmpeg/ffprobe, sox, and Python 3 with numpy,
scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. You can view
still images you extract.
Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 6 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, MIT.
One run per agent.
The agent gets a 90-minute simulated drive recorded by four roof cameras and a four-microphone array, plus a chapter of a traffic bylaw. For every emergency vehicle that sounds its siren it reports when the siren is first heard and from which compass direction, when and in which camera the vehicle first comes into view, what kind of vehicle it is, and whether the car yielded as the bylaw requires. The reference answer holds 31 vehicles.
What a good answer has to get right, and how the verifier checks it
- Every siren vehicle is found, and none is invented
- The score is an F1 over vehicles with one-to-one matching. A missed vehicle lowers recall; an entry that matches no vehicle, or cannot be read, lowers precision.
- The siren is timed and its direction heard
- The time the siren is first audible must be within 0.5 s, on the video’s timeline, and the direction must be the exact one of eight compass sectors. The microphone recording has its own clock, so its offset from the video has to be recovered first.
- The vehicle is seen
- The camera must be one in which the vehicle first comes into view, and the time within 1.0 s of that camera’s first-view time. In view means at least 3000 pixels of the vehicle visible for a second. Where two cameras pick it up less than a second apart, either is accepted with its own time.
- The vehicle is identified
- The type must be exact: police car, ambulance or fire engine.
- The bylaw is applied
- The verdict must be exact: yielded, failed to yield, or not required. It is computed from the simulator’s log under the rules of the bylaw chapter given to the agent.
How the checks become one score
Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 31 vehicles. Every vehicle counts the same, and an entry counts only when all of its fields are right: a vehicle heard at the right moment with the wrong verdict is a miss and a false positive.
What scores zero
No siren_ledger.json, a file that cannot be read or whose “vehicles” is not a list, an empty list, or a list in which no entry matches a vehicle on every field.
Outside the score: 124 further siren vehicles in the drive that the audit judged not observable from the recordings. They are excused: an unmatched entry whose siren time is within 1.5 s of one is left out of the count, so reporting one neither helps nor costs. The verifier also reports per-field agreement as diagnostics that do not enter the score.
Verifier code
Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.