AgenticVBench-Omni

AgenticVBench-Omni / Scientific observation

Aviary

aviary-caller-attribution-ledger

You are given 21 one-minute recordings of a group of eight zebra finches living together in an indoor aviary. The birds often call at the same time. For each recording, find every vocal bout of every bird: which bird it was, when the bout started and when it ended.

Family
Scientific observation
Inputs
videoaudiodocuments
Media
21 videos, 21 min, no sound track; 21 audio files, 21 min
Reference answer
143 scored bouts
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matchingmean of components

Task package · verifier code

Instruction

Full instruction given to the agent

Aviary caller ledger

You are given 21 one-minute recordings of a group of eight zebra finches living together in an indoor aviary. The birds often call at the same time. For each recording, find every vocal bout of every bird: which bird it was, when the bout started and when it ended.

Materials (all in the working directory)

  • clips/clip01.mp4 … clips/clip21.mp4 — video, 60 s each, 2976×2016 pixels, 390625/8192 ≈ 47.68 frames per second, no audio track.
  • clips/clip01.flac … clips/clip21.flac — 7-channel microphone audio for the same 60 s, 16-bit, 24414 Hz. Audio and video come from one hardware clock: audio sample 0 is at video frame 0, and one video frame spans 512 audio samples.
  • roster.json — for each clip, the birds present. Each bird wears a small coloured backpack on its back and is named by that colour: purple, red, peach, blue, green, white, yellow, brown. A colour always means the same bird.
  • backpacks.png — one example of each backpack colour, cropped from the top camera.

The clips are in random order and come from several days.

The aviary

The birds live in a box with a 90 cm × 55 cm floor. The lower 40 cm (the "stage") holds perches, food, water, a sand bath and two nest boxes (front-left and rear-right corners, 30–40 cm up); above it a tent-shaped upper part narrows towards the ceiling. Birds can be in the upper part or hidden behind one another.

Each video frame is a composite of camera views. Rectangles are given as left x, top y, width × height, in pixels:

region view
1000, 726, 1976×1290 top camera, looking straight down on the stage
1000, 0, 1976×726 back camera, through the rear wall (mirror image)
220, 726, 780×1290 left camera, through the left wall (mirror image, rotated 90°)
0, 0, 1000×726 two cameras looking into nest boxes (greyscale)

The pixels at 0, 726, 220×1290 are unused.

Audio channels (1-based):

channel microphone position
1 centred above the rear wall of the stage
2 centred above the left wall of the stage
3 centred above the front wall of the stage
4 centred above the right wall of the stage
5 ceiling of the enclosure, next to the top camera
6 inside the front-left nest box
7 inside the rear-right nest box

Definitions

  • Vocalization: any sound a bird makes with its voice — song syllables and introductory notes, every kind of call (short soft calls, loud distance calls, begging calls), and young birds' unfinished, babbling song. Not vocalizations: wing beats, take-offs and landings, pecking, seed husking, bill wiping, scratching, and noises from perches, feeders or walls.
  • Bout: take one bird's vocalizations in the clip in time order. Two consecutive vocalizations of that bird belong to the same bout when the silence from the end of the first to the start of the next is shorter than 0.5 s; otherwise the second one starts a new bout. A single isolated call is a bout on its own. A bout starts at the start of its first vocalization and ends at the end of its last. Only the bird's own sounds count: another bird's vocalization never joins or splits this bird's bouts, and when two birds vocalize at the same time each of them has its own bout.
  • Times are seconds from the start of the clip (a number such as 12.43).
  • A listed bout is correct when the clip and the bird are right and its start and its end are each within 0.1 s of the true bout's start and end. A bout attributed to the wrong bird, one bout split into several, and several bouts merged into one are all wrong. Listing a bout that did not happen is wrong; missing one is also wrong.
  • One bird per sound. Do not cover several possibilities by listing the same sound under more than one bird. If two bouts you list for different birds in the same clip have starts within 0.6 s of each other and ends within 0.6 s of each other, then unless both of them are correct, both count as wrong. Two birds really vocalizing together is fine when you name both correctly.

How much the birds vocalize differs a lot between clips: some clips have only a few bouts, others more than a hundred.

What to submit

Write output/bouts.json in exactly this shape:

{
  "bouts": [
    {"clip": "clip01", "bird": "red", "start": 12.40, "end": 12.48},
    {"clip": "clip01", "bird": "white", "start": 12.45, "end": 14.02}
  ]
}
  • One entry per bout, in any order; bird must be a colour listed for that clip in roster.json.
  • Write this file early and keep overwriting it as you complete clips. There is a time limit, and whatever is in the file when the session ends is what gets graded.

Rules

  • Work only inside the working directory. Do not read, write or search outside it.
  • There is no internet access and none is needed; pip install will not work. Do not rely on memory of this or any other recording. Every entry must come from what you observe in these files.
  • Available tools are what is installed: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 44 files (environment/media/SHA256SUMS.txt). Licences named for this task: GPL-2.0-or-later, mit.

Score

AgentScore (0 to 1)
GPT-6-Astra0.066
Claude Opus 5.50.000
Claude Opus 50.005
Gemini 3.8 Flash0.004
DeepSeek 4.1 Flash0.005

One run per agent.

The agent gets 21 one-minute recordings of eight zebra finches in an aviary (video, 7-channel microphone audio and a roster of the birds) and lists every vocal bout of every bird: clip, bird, start and end. Seven of the clips are scored, the ones whose minute has manual vocal annotation in the source dataset. The reference holds 143 scored bouts.

What a good answer has to get right, and how the verifier checks it

Each bout is given to the bird that made it
A listed bout can only count for a true bout of the same clip and the same bird. The 143 scored bouts are the contested ones: each overlaps sound from another bird.
Each bout is timed
Its start and its end must each be within 0.1 s of the true bout’s, with one-to-one matching. A bout split in two, or two bouts merged into one, therefore matches nothing.
Bouts are neither invented nor missed
Each scored clip and bird is one ledger with its own F1 = 2 × hits / (2 × hits + wrong entries + missed bouts).
One bird per sound
Two entries for different birds in the same clip whose starts are within 0.6 s of each other and whose ends are within 0.6 s of each other are treated as one sound listed twice. Unless both are correct, both count as wrong. The instruction states this rule.
Uncertain sound is not held against the answer
An entry that is not a hit is left out of the count when it overlaps sound whose caller the annotation does not know, overlaps a bout of the same bird whose boundaries are not stable, or matches an uncontested bout of the same bird within 0.1 s. A wrong bird on an uncontested bout is still wrong.

How the checks become one score

Score = the mean of the ledger F1 values, over every scored clip and bird that has at least one scored bout or one counted entry. Each ledger weighs the same, whether the bird has two bouts in that clip or sixty.

What scores zero

No bouts file or one that cannot be read, or an answer in which no entry matches a scored bout of the right bird within 0.1 s at both ends.

Outside the score: everything listed for the 14 clips that are not scored, and the 250 annotated items the reference marks as don’t-care, which never count as missed.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
tolerance_s0.1value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.07
Wall time
40.2 min
Steps shown
93

Loading the run…