AgenticVBench-Omni

AgenticVBench-Omni / Transport and operations

Dual Runway

dualrunway-ops-ledger

You are given two files covering the same real stretch of time at a real airport's dual-runway complex, recorded from a camera near the runways:

Family
Transport and operations
Inputs
videoaudio
Media
1 video, 2 h 34 min, no sound track; 1 audio file, 2 h 36 min
Reference answer
49 operations
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

Runway Operations Ledger

You are given two files covering the same real stretch of time at a real airport's dual-runway complex, recorded from a camera near the runways:

  • /workspace/materials/runway.mp4 — silent video (no audio track).
  • /workspace/materials/tower.mp3 — the tower radio audio for the same period.

The two files are not time-aligned — they do not share a start time or a clock, and no metadata anywhere tells you the offset between them. You have to work it out from their content: match the sequence and spacing of the operations you can see against the sequence and spacing of the clearances you can hear.

Two rectangular regions of the frame are blanked to black for the whole video. Nothing you need is inside them; do not spend time on them.

The footage is at night, and the camera is a moving PTZ camera operated by a person: it pans and zooms to follow individual aircraft, and it also wanders off to look at the city between aircraft. It does not sit still on the runway, and it does not catch every aircraft that uses it.

Reconstruct the ledger of the runway operations in the video — every aircraft that either lands (main gear touches down) or takes off (lifts off during the departure roll).

The ledger covers the whole recording. The video runs over two and a half hours and the traffic is steady throughout: there are dozens of operations, spread from the first minutes to the last. A ledger built from the opening stretch and then declared finished is not a ledger — plan your time so that every part of the recording gets looked at.

For each operation, report:

  • video_time — the timestamp in the video file's own timeline (hh:mm:ss from the start of runway.mp4): for a landing, the touchdown; for a takeoff, the liftoff.
  • operation — exactly one of: landing, takeoff.
  • aircraft_type — the aircraft family, from this closed vocabulary only (pick the closest match; do not invent new labels): A220, A320-family, A330, A340, A350, A380, B737-family, B747, B757, B767, B777, B787, regional-jet, other.
  • callsign — the ICAO callsign (airline 3-letter code + flight number, e.g. SWA1234) as spoken by the tower controller in that aircraft's clearance ("cleared to land" for a landing, "cleared for takeoff" for a takeoff). Controllers speak the airline's radiotelephony name, not the ICAO code — "Southwest twelve thirty-four" is SWA1234. Use this table to convert:
telephony (spoken) ICAO code telephony (spoken) ICAO code
Southwest SWA American AAL
Delta DAL United UAL
Alaska ASA JetBlue JBU
Spirit NKS Frontier FFT
Allegiant AAY Sun Country SCX
SkyWest SKW Horizon Air QXE
Envoy ENY Republic RPA
Mesa ASH Endeavor EDV
Jet Set JSX Breeze MXY
Rouge ROU WestJet WJA
Air Canada ACA Copa CMP
Aeromexico AMX Volaris VOI
Speedbird BAW KLM KLM
Lufthansa DLH Air France AFR
FedEx FDX UPS UPS

A carrier not in this table: transcribe the ICAO code as best you can determine it from what is said, and do not guess a carrier that was not actually spoken. An aircraft with no airline callsign (business jets, light aircraft) is addressed by its registration — report that as spoken, e.g. N123AB.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "operations": [
    {"video_time": "00:14:32", "operation": "landing", "aircraft_type": "B737-family", "callsign": "SWA1428"},
    {"video_time": "00:19:05", "operation": "takeoff", "aircraft_type": "A320-family", "callsign": "AAY518"}
  ]
}
  • One entry per runway operation, in any order.
  • video_time tolerance: within about 45 seconds of the real moment.
  • Write this file early and keep overwriting it. As soon as you have even one complete operation, write solution.json with what you have, and rewrite it each time you complete another. There is a time limit; whatever is in the file when the session ends is what gets graded, so never leave your findings only in notes or in intermediate files.
  • Budget your time as though the session could end at any moment. The expensive mistake here is to process the whole video, then the whole audio, then pair the two at the end: that ordering produces nothing gradeable until the very last step, and it loses everything if you run out of time. Working through the recording in chronological stretches — see a stretch, hear the same stretch, pair it, append what you confirmed — keeps a usable answer on disk the whole way through.
  • An entry with a missing or empty callsign counts against you exactly like a wrong one. If you cannot name an aircraft, leave that operation out entirely rather than submitting it with the field blank.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • No internet access is available and none is needed. Do not rely on memory or general knowledge of any specific airport, carrier schedule, or route network — every entry must be justified by something you directly observed in runway.mp4 or heard in tower.mp3. Do not guess.
  • Count only completed runway operations: landings (touchdown) and takeoffs (liftoff). Do not report go-arounds, rejected takeoffs, aircraft only taxiing or holding, or aircraft that merely cross a runway.
  • Tools in the image include ffmpeg/ffprobe for seeking and sampling the video and for cutting audio segments, and a local speech-to-text model at $WHISPER_MODEL_DIR (load it with faster_whisper.WhisperModel in Python) for transcribing tower.mp3 — there is no other way to recover the audio content.
  • There is no network, so pip install will not work. Already installed and sufficient for everything here: numpy, av (PyAV — decodes video frames directly into numpy arrays), and faster_whisper. PIL, cv2 and scipy are not installed; do not plan around them.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.556
Claude Opus 5.50.435
Claude Opus 50.308
Gemini 3.8 Flash0.362
DeepSeek 4.1 Flash0.101

One run per agent.

The agent gets silent camera video of a dual-runway complex and a separate recording of the tower radio for the same stretch of time, with no shared clock. It lists every runway operation: its time in the video, landing or takeoff, the aircraft type and the callsign. The reference answer holds 49 operations that could be verified in the picture.

What a good answer has to get right, and how the verifier checks it

Every operation is found, and none is invented
The score is an F1 over operations; each reference operation can be matched once. A missed operation lowers recall and an entry that matches nothing lowers precision.
What is seen is paired with what is heard
An entry counts only when operation, aircraft type and callsign are all right. The callsign exists only in the radio audio and the other fields only in the picture, so a correct entry requires working out the offset between the two files.
The aircraft type is named
The type must be the exact one of 14 listed families, such as A320-family, B737-family or regional-jet.
The operation is placed in time
The video time must be within 45 s. The tolerance is wide on purpose: the reference times come from ADS-B reports that are about 22 s apart around touchdown, and five operations pinned by hand against the frames left about 15 s of scatter.
Real operations the camera did not show are not held against the answer
161 further operations can be heard on the radio but were not tracked by the camera, so they cannot be confirmed in the video. An entry with the right operation and callsign within 120 s of one is left out of the count; it neither helps nor costs.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 49 verified operations. Every operation counts the same, and there is no credit for an entry that is right on some fields: an operation with the right time and type and no callsign counts as wrong.

What scores zero

No solution.json, a file that cannot be read or whose “operations” is not a list, an empty list, or a list in which no entry is right on all four fields.

Outside the score: the 161 unconfirmed operations, the runway used, which the tower states in nearly every clearance, and entries whose video time cannot be parsed, which are skipped. The verifier also reports how many entries are right on fewer fields, as diagnostics.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
DONT_CARE_TOL120constant in the verifier code
video_time_tolerance_s45value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.56
Wall time
37.1 min
Steps shown
136

Loading the run…