AgenticVBench-Omni

AgenticVBench-Omni / Scientific observation

Siberian Jay

siberian-jay-feeder-individual-ledger

videos/ holds four real field recordings from a long-term study of wild Siberian jays in northern Sweden: feeder_1.mp4 … feeder_4.mp4. Each is a single continuous take from a fixed camera pointed at a wooden feeding perch: an upright post with two horizontal poles, each with a bundle of food wrapped in light-coloured cloth tied at the middle. Jays fly in, perch, peck at the food bundles, and leave, often several at once. The files are 1920×1080 at 25 fps, 15 to 31 minutes long, with no audio. Fast motion shows interlacing artifacts.

Family
Scientific observation
Inputs
videodocuments
Media
4 videos, 1 h 45 min, no sound track
Reference answer
104 bouts
Deliverable
/workspace/output
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

Siberian Jay Feeder Ledger

videos/ holds four real field recordings from a long-term study of wild Siberian jays in northern Sweden: feeder_1.mp4 … feeder_4.mp4. Each is a single continuous take from a fixed camera pointed at a wooden feeding perch: an upright post with two horizontal poles, each with a bundle of food wrapped in light-coloured cloth tied at the middle. Jays fly in, perch, peck at the food bundles, and leave, often several at once. The files are 1920×1080 at 25 fps, 15 to 31 minutes long, with no audio. Fast motion shows interlacing artifacts.

Identifying birds

Each jay carries four leg rings, two on each leg, one above the other. A bird's code is four letters in this order: top ring on its left leg, bottom ring on its left leg, top ring on its right leg, bottom ring on its right leg. Left and right are the bird's own left and right, not the image's (a bird facing the camera shows its left leg on the image's right).

letter ring colour letter ring colour
a aluminium (metal) o orange
c chrome (grey) y yellow
g green s sky (light blue)
l lime (light green) b blue
r red m mauve (purple)
w white p pink

Each video's list below names every ringed bird that could appear in it. Not every listed bird appears.

video candidate birds
feeder_1.mp4 arlb, cgba, cyap, gagr, gamg, gaob, garw, mamo, mays, opma, yaws
feeder_2.mp4 bowa, camc, gamb, mwwa, ooca, oyca, sswa, wpap, wsca, yamp, yaro, ypca
feeder_3.mp4 arsg, bpay, gabs, gacm, gagp, gamm, garr, gaym, gays, gayw, gmwa, lgwa, mapm, mawr, oarl, occa, oowa, ppca, roca, rwas, sbca, ssca, yagg, yass, yoca
feeder_4.mp4 arsg, bpay, brca, bsma, bwya, byca, cawy, gmwa, mawr, ppas, roca, rwas

Example: a bird coded oaor has orange over aluminium on its own left leg and orange over red on its right leg. A bird whose rings you cannot match to its video's list (for example an unringed bird) is ignored.

Task

For every listed bird that feeds in each video, reconstruct every feeding bout as the list of its pecks.

  • Peck: one dip of the head to the food. The bird lowers its head until its bill reaches a food bundle, then raises it again. One dip is one peck, however long the bill stays at the food and however many small nibbling or tugging movements it makes there; the next peck starts only after the head has come back up. Touching the perch, the post or the bird's own feet is not a peck. A peck's time is the first frame in which the bill touches the food, given in seconds from the start of that video file (frame index ÷ 25).
  • Feeding bout: take all pecks by one bird in one video, in time order, and split them wherever two consecutive pecks are 20 seconds or more apart. Each resulting group is one bout; a lone peck is a bout of one. Pecks by different birds never share a bout, and two birds feeding at the same time each have their own bouts.

Output

Write output/bouts.json:

{
  "bouts": [
    {"video": "feeder_2.mp4", "bird": "gamb", "peck_times_s": [1500.2, 1501.9, 1503.0, 1506.4]},
    {"video": "feeder_4.mp4", "bird": "byca", "peck_times_s": [640.0]}
  ]
}
  • One entry per bout, any order. video must be one of the four files and bird must be on that video's list. peck_times_s lists every peck of the bout.
  • An entry matches a real bout only if all of these hold: the bird is right; its first and last peck times are each within 2 seconds of the real bout's first and last pecks; its number of pecks is within 15% of the real count (at least ±1); and at least 80% of the real pecks can each be paired with a different listed peck time within 0.3 seconds. A bout split or merged differently from the definition above does not match.
  • Every entry you submit counts: entries that match no real bout count against you, and so does listing the same bout under several birds.
  • Write the file early and keep overwriting it as you go. There is a time limit, and only what is in output/bouts.json when the session ends is graded.

Rules

  • Work only inside this working directory. Do not read, write or search outside it.
  • There is no internet access. Do not rely on memory of these birds, this study or any published annotations; every entry must come from what you observe in the videos.
  • Available tools are what is installed: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless and matplotlib. pip install will not work. You can extract frames or crops and look at them as images.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 4 files (environment/media/SHA256SUMS.txt). Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.279
Claude Opus 5.50.067
Claude Opus 50.010
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

This task’s ground truth is not in the repository: it is derived from a dataset whose licence does not allow publishing it. Reviewers find it in the media dataset; steps/solve/tests/ground_truth/README.md in the task package says where it goes and how to check it.

The agent watches four field recordings of wild Siberian jays at a feeding perch, 15 to 31 minutes each and without sound, and lists every feeding bout: the video, the bird, read from its four leg rings, and the time of each peck. The reference holds 104 bouts by 12 feeding birds.

What a good answer has to get right, and how the verifier checks it

Each bout is given to the right bird
An entry can only count for a true bout of the same video and the same bird. Each video’s list of candidate birds mixes the birds that feed there with ringed birds that never do, so spreading one bout over several candidates costs as much as it can gain.
The bout is bounded
Its first and last peck times must each be within 2 s of the true first and last peck.
The pecks are counted
The number of pecks must be within 15% of the true count, and never tighter than one peck either way.
The pecks are located
At least 80% of the true pecks must each pair with a different listed peck time within 0.3 s. A list of evenly spaced times, or every motion peak, does not pass.
Bouts are neither invented nor missed
Every entry in the file counts as submitted. True bouts of one bird start at least 20 s apart, so an entry can match at most one.

How the checks become one score

Score = matched bouts / the larger of 104 and the number of entries submitted. A bout either meets all four conditions or counts for nothing; there is no credit for a bout with the right bird and bounds but poorly placed pecks.

What scores zero

No bouts.json or one that cannot be read, or an answer in which no entry meets all four conditions for a true bout.

Outside the score: nothing in the answer is ignored. A bout split into two entries, or two bouts merged into one, counts as wrong.

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.28
Wall time
163.7 min
Steps shown
664

Loading the run…