AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

Hidden Sound

hidden-sound-source-map

Your working directory contains:

Family
Spatial measurement and tracking
Inputs
videoaudiodocuments
Media
1 video, 30 min, with sound
Reference answer
24 sources with 98 on-intervals
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Hidden sound sources in a house

Your working directory contains:

  • walkthrough.mkv — a 30-minute recording of a person walking slowly through a single-storey house. The video (1280×720, 15 fps) comes from a camera worn at eye level. The audio is a lossless 2-channel binaural recording (48 kHz): one microphone in the person's left ear, one in the right ear. Video and audio share one timeline.
  • floorplan.png and floorplan.json — the layout of the house in metres: rooms, open doorways (there are no doors), and windows.

Somewhere in the house there are between 15 and 30 fixed sound sources. Each one makes a continuous or repeating sound while it is switched on. Several sources make similar sounds, and no two sources play exactly the same recording. Nothing in the video shows where a source is: the sources are not visible, and where they are has nothing to do with the furniture or with what they sound like. Each source stays at one fixed position for the whole recording. Several sources can be in the same room, and some rooms may have none.

Every source is switched on and off several times during the recording, often while the camera is in another room. Some are already on when the recording starts, and some are still on when it ends.

Find every source. For each one, report the room it is in, its position on the floor plan, and every time interval during which it was on.

Conventions

  • Position: floor-plan coordinates in metres — x increases to the east, y increases to the north, origin at the south-west outer corner of the house (exactly as in floorplan.json and the axes of floorplan.png). Report the horizontal position (x, y) of the sound source; its height is not needed.
  • Room: exactly one of living_room, kitchen, laundry, hallway, bedroom_1, bathroom, bedroom_2, study — the room rectangles in floorplan.json.
  • Camera: horizontal field of view 90° (vertical about 58.7°), no lens distortion, optical axis always level (no pitch or roll), lens centre 1.6 m above the floor. The camera turns with the head: the two ear microphones lie on a horizontal line perpendicular to the viewing direction, about 0.15 m apart, left ear on the left side of the image.
  • Time: seconds from the start of walkthrough.mkv (0 to 1800).
  • On-interval: a maximal stretch of time during which the source is producing its sound, from the moment it is switched on to the moment it is switched off. A source already on when the recording starts has an interval starting at 0; one still on at the end has an interval ending at 1800. For a sound made of repeated short events, the source is on for the whole stretch over which the repetition continues, including the short gaps between events. Every on-period and every off-period of a source lasts at least 40 s, and every switch-on and switch-off happens at a moment when that source can be heard at the camera.

What to submit

Write output/sources.json in exactly this shape:

{
  "sources": [
    {"room": "kitchen", "x": 7.9, "y": 1.2, "on": [[0.0, 412.5], [980.0, 1105.0], [1600.0, 1800.0]]},
    {"room": "study", "x": 12.4, "y": 8.1, "on": [[655.0, 1210.0]]}
  ]
}
  • One entry per source, with all of its on-intervals.
  • What is graded is the list of on-intervals. An on-interval you report counts as correct only if all of these hold: its source's room is exactly right, its source's (x, y) is within 1.0 m of the true position, its start is within 5 s of the true switch-on time, and its end is within 5 s of the true switch-off time. Missed intervals and wrong or extra intervals both count against you; a source reported in the wrong room or more than 1.0 m away earns nothing for any of its intervals.
  • Write the file early and keep updating it as you learn more. Whatever is in the file when the session ends is what gets graded.

Rules

  • Work only inside this working directory. There is no internet access, and none is needed; do not rely on memory of any particular recording.
  • Tools available: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. Nothing else can be installed (no network).

Media

Frame from walkthrough.mkv
walkthrough.mkv · 450 s
Frame from walkthrough.mkv
walkthrough.mkv · 900 s
Frame from walkthrough.mkv
walkthrough.mkv · 1350 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 3 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, CC0 1.0, GPL-3.0-or-later, Public Domain.

Score

AgentScore (0 to 1)
GPT-6-Astra0.178
Claude Opus 5.50.016
Claude Opus 50.024
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets a 30-minute first-person walk through a house, recorded with a microphone in each ear, and the floor plan. Hidden sources switch on and off around the house and none is visible. For each source the agent reports its room, its position on the plan and every interval during which it was on. The reference answer holds 24 sources with 98 on-intervals.

What a good answer has to get right, and how the verifier checks it

Each source is placed in the right room
A reported source can only be paired with a true source whose room label is identical, out of eight rooms.
Each source is located on the plan
Its position must be within 1.0 m of the true position, measured on the floor plan. Sources are paired one to one, in the way that matches the most intervals.
Each switch-on and switch-off is timed
An on-interval counts when its start and its end are each within 5 s of a true on-interval of the paired source.
Intervals are neither invented nor missed
The score is an F1 over on-intervals, not over sources, so a source that switches on five times weighs five times as much as one that is on once.

How the checks become one score

Score = F1 = 2 × matched intervals / (reported intervals + 98). An interval counts only when its source has the right room and position and both its ends are timed: a correctly timed interval attached to a source placed 2 m away counts as wrong.

What scores zero

No sources.json or one that cannot be read, no intervals at all, more than 100 sources or 2000 intervals, or an answer in which no interval matches.

Scores from this verifier: the reference answer with one thing changed

SubmissionScorePrecisionRecall
Careful work
The reference answer1.0001.0001.000
0.5 m and 2 s offEvery source is placed 0.5 m to one side and every switch time is 2 s late.1.0001.0001.000
One kind of mistake
Half of the sourcesEvery second source is left out; the others are correct.0.6761.0000.510
Only the first interval of each sourceEach source is located correctly, and only its first on-interval is reported.0.3931.0000.245
Right room, 1.5 m offEvery source is placed 1.5 m to one side, with the right room label. Times are correct.0.0000.0000.000
Switch times 10 s lateRooms and positions are correct; every switch-on and switch-off is reported 10 s late.0.0000.0000.000
One interval per sourceEach source is reported as on from its first switch-on to its last switch-off, with the gaps ignored.0.0000.0000.000
No real work
One source per room, always onOne source at a fixed point in each of the eight rooms, on for the whole recording.0.0000.0000.000
No file0.0000.0000.000
An agent’s answer
A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step0.1780.1640.194

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the kind of sound each source makes, and the height of the source; only its position on the plan is compared.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
POS_TOL_M1constant in the verifier code
TIME_TOL_S5constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.18
Wall time
66.5 min
Steps shown
129

Loading the run…