AgenticVBench-Omni

AgenticVBench-Omni / Scientific observation

Pigeon Flock

pigeon-flock-mocap-ledger

The working directory holds six recording sessions of a group of homing pigeons walking freely on a burlap mat laid on the floor of a motion-capture hall. Each session was filmed by four synchronized cameras standing around the mat. For every session, report where each bird was once per second, and every time one bird displaced another.

Family
Scientific observation
Inputs
videodocuments
Media
24 videos, 4 h 8 min, no sound track
Reference answer
3,481 positions and 32 displacements
Deliverable
/workspace/output
Scoring
F1 / precision–recallgeometric (IoU / distance)one-to-one matchingweighted

Task package · verifier code

Instruction

Full instruction given to the agent

Pigeon Flock Ledger

The working directory holds six recording sessions of a group of homing pigeons walking freely on a burlap mat laid on the floor of a motion-capture hall. Each session was filmed by four synchronized cameras standing around the mat. For every session, report where each bird was once per second, and every time one bird displaced another.

Materials

  • sessions/S1/view1.mp4 … sessions/S6/view4.mp4: 1920×1080, 30 frames per second, no audio. Within a session the four files start at the same instant and have the same number of frames; frame n (counting from 0) of any view of that session is time n / 30 seconds.
session birds frames whole seconds N
S1 10 16641 554
S2 10 18260 608
S3 10 25457 848
S4 10 16640 554
S5 10 16821 560
S6 5 17540 584
  • The cameras' positions, orientations and focal lengths are not given. They do not move during a session but differ between sessions, and view1 … view4 do not name the same camera in different sessions. Lens distortion has been removed from the images (ideal pinhole cameras). The images are real footage, not renders.
  • roster/roster.json: the birds present in each session. Birds have codes P01 … P18; a code always means the same pigeon, in every session.
  • roster/S1/P03_view2_1.jpg etc.: reference crops of every bird of every session, cut from that session's cameras at moments that are not part of the videos. The background around the bird has been replaced by flat grey; parts of a neighbouring bird can remain. File name = bird, view of that session, index.
  • reference_points.json: for each session, 12 entries {"frame": n, "x": …, "y": …, "z": …}. At that frame of that session, the centre of the backpack of one of the birds (which one is not said) was at that point, in metres, in the arena frame below.

Birds

Every pigeon wears a small black backpack on its back and small markers on its head. All backpacks look alike. Each backpack also carried a printed label, which has been blurred out wherever it could be read (in the videos and in the reference crops), so on birds near a camera the backpack shows as a blurred patch on the back. Tell birds apart by their plumage. Birds sometimes fly short distances or step off the mat.

Arena frame

Right-handed, metres, with z up from the floor. The x axis is parallel to the straight dark tape line on the floor that runs under the mat and shows on both sides of it, and points toward the side of the mat where a small yellow electronics box with a red battery pack stands on the floor. y is 90° counter-clockwise from x seen from above. The origin is on the tape line, roughly midway between the two long edges of the mat. The frame is the same hall-fixed frame in every session (the mat itself was laid a little differently each day); reference_points.json gives it exactly.

What to report

Position. A bird's position is the point on the floor directly below the centre of its backpack, given as [x, y]. For each session, each bird of that session and each whole second t = 0, 1, …, N − 1, report the bird's position at frame 30·t, or null when you do not give one. A position counts as correct when it is within 0.12 m of the true point. A position listed under the wrong bird is wrong.

Displacement. Write p_A(f) for bird A's position at frame f, and d(f) for the distance between A and another bird B at frame f. Actor A displaces target B at frame f0 when all of these hold:

  1. d(f0) ≤ 0.35 m, and d(f) > 0.35 m for every frame f from f0 − 30 to f0 − 1 (they were apart for the whole second before);
  2. A moved at least 0.08 m over that second, |p_A(f0) − p_A(f0 − 30)| ≥ 0.08 m, and B moved at most 0.15 m, |p_B(f0) − p_B(f0 − 30)| ≤ 0.15 m;
  3. within the next 2 seconds B gets at least 0.25 m away from where it was at f0: |p_B(f) − p_B(f0)| ≥ 0.25 m for some frame f from f0 to f0 + 60.

Its time is f0 / 30 seconds. Only contacts with the full second before and 2 seconds after inside the video count. Approaches that meet 1 and 2 but not 3 are not displacements; do not list them. A listed displacement is correct when actor and target are right and its time is within 1.5 s of the true time. Listing a displacement that did not happen is wrong, missing one is wrong, and each true displacement can be matched only once, so a duplicate or an actor/target swap is an extra wrong entry.

Output

Write output/ledger.json:

{
  "sessions": {
    "S1": {
      "positions": {
        "P03": [[0.412, 1.870], [0.415, 1.861], null],
        "P11": [[-1.203, 0.334], [-1.190, 0.340], [-1.150, 0.402]]
      },
      "displacements": [
        {"actor": "P03", "target": "P11", "t": 123.4}
      ]
    }
  }
}

The values only illustrate the format. positions maps each bird code to a list whose item t is the position at second t (a list of N items for that session; shorter lists are read as null for the missing seconds). There is a time limit, and whatever is in output/ledger.json when the session ends is what counts. Write the file early and keep updating it as you work.

Rules

  • Work only inside this directory. There is no internet access. Do not rely on memory or outside knowledge of any recording; everything must come from the files here.
  • Available tools: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. Nothing else can be installed.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 466 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, MIT.

Score

AgentScore (0 to 1)
GPT-6-Astra0.326
Claude Opus 5.50.689
Claude Opus 50.092
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches six sessions of homing pigeons walking on a mat, each filmed by four synchronized cameras, and writes where each bird was once per second and every time one bird displaced another (time, actor, target). The reference, from motion capture, scores 3,481 position seconds and 32 displacements.

What a good answer has to get right, and how the verifier checks it

Each bird is located
A position counts when it is within 0.12 m of the motion-capture position for that bird at that second. A position listed under the wrong bird is wrong.
Identity is kept when birds come together
The scored seconds are the contested ones: the bird was within reach of another bird shortly before and was moving. On these, a missing position is a miss, and a wrong one is both a miss and a false positive.
Wrong positions cost everywhere
On any other second with a reference position, a wrong entry is a false positive and a right one adds nothing. Leaving a second empty there costs nothing.
Every displacement is found, and none is invented
A listed displacement counts for a true one of the same session, actor and target when its time is within 1.5 s, one to one. The task defines a displacement by distances and movements over three seconds, so it follows from the positions.
Borderline displacements are not held against the answer
An unmatched displacement is left out of the count when it is within 1.5 s of an approach with the same actor and target that falls just short of the definition’s thresholds.

How the checks become one score

Score = 0.5 × position F1 + 0.5 × displacement F1, each F1 = 2 × hits / (2 × hits + false positives + misses). The two halves weigh the same although there are far more position seconds than displacements.

What scores zero

No ledger or one that cannot be read, or an answer with no position within 0.12 m on a scored second and no displacement matched.

Outside the score: seconds for which the reference has no usable position, and the borderline approaches named above.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
tol_m0.12value the verifier reports using
tol_s1.5value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.33
Wall time
34.7 min
Steps shown
89

Loading the run…