AgenticVBench-Omni

AgenticVBench-Omni / Scientific observation

Savanna Drone

savanna-drone-behaviour-budgets

The working directory holds 14 silent drone video clips of wild Grevy's zebras, plains zebras and reticulated giraffes on an African savanna, filmed from a few tens of metres up:

Family
Scientific observation
Inputs
videodocuments
Media
14 videos, 49 min, no sound track
Reference answer
274 bouts of 28 animals
Deliverable
/workspace/output
Scoring
F1 / precision–recallmean of components

Task package · verifier code

Instruction

Full instruction given to the agent

Drone behaviour budgets

The working directory holds 14 silent drone video clips of wild Grevy's zebras, plains zebras and reticulated giraffes on an African savanna, filmed from a few tens of metres up:

  • clips/clip_01.mp4 … clips/clip_14.mp4 — 3840×2160, 30000/1001 fps (29.97), no audio, 2 to 5.5 minutes each, about 49 minutes in total.
  • marks/clip_NN.json — the animals you must follow in that clip. Each is given by a letter and its bounding box [x0, y0, x1, y1] in video pixels on the first frame (frame 0) of the clip.
  • marks/clip_NN_first_frame.jpg — frame 0 of the clip with the same boxes drawn and lettered.

28 animals are marked across the 14 clips (1 to 5 per clip). The drone is flown by hand: it drifts, pans, tilts and zooms. The animals graze, stand, walk and sometimes run; they bunch together, cross paths, pass behind bushes, and share the picture with unmarked animals of the same species. A marked animal stays the same individual for the whole clip; you have to keep track of which one it is from frame 0 to the last frame.

Question

For every marked animal, give its behaviour timeline across the whole clip and its total seconds per behaviour.

Behaviours (closed vocabulary)
label meaning
walk moving at a walk
trot moving at a trot (a faster, two-beat gait)
run moving at a canter or gallop
graze eating grass or other vegetation on the ground
browse eating from trees or bushes
head_up head raised, looking around or watching its surroundings
auto_groom grooming itself: licking, scratching, nibbling or rubbing its own body
not_visible the animal cannot be judged: hidden by vegetation or other animals, too blurred, or outside the picture
other visible but doing something not listed above (e.g. defecating, urinating, sniffing, lying down, dust-bathing, grooming another animal, fighting)
Timeline
  • Times are seconds from the start of the clip: frame n begins at n × 1001 / 30000 s. The timeline for an animal runs from 0 to the clip's duration (its frame count × 1001/30000).
  • A bout is a stretch of one behaviour lasting at least 2.0 s. A change lasting less than 2.0 s is not a bout; it belongs to the bouts around it. Report bouts, not frame-level flicker. A stretch lasting 2.0 to 4.0 s may be reported as its own bout or folded into the bouts around it; either is accepted.
  • Consecutive bouts of the same behaviour separated only by not_visible or other time are one bout. Whatever you report for time in which the animal truly cannot be judged, or is truly doing something other, is not checked.
  • A reported bout is correct when its behaviour matches and its start and its end are each within 1.5 s of the true bout's start and end. Where the true boundary sits next to time in which the animal cannot be judged, that whole span is accepted for the boundary.
  • Segments must not overlap. Gaps between your segments are treated as not_visible.
Time budget

budget_seconds gives, for each of the seven behaviours walk, trot, run, graze, browse, head_up, auto_groom, the total seconds the animal spends in it over the clip, counted from its bouts as defined above (not_visible and other time is excluded). A value is correct when it is within 10 % of the true total, or 3 s, or 1.5 s × √n where n is the number of places that behaviour's true bouts meet a different behaviour, whichever is largest. Give 0 for behaviours the animal never shows; reporting more than 3 s for a behaviour the animal never shows counts as a wrong value.

What to submit

Write output/behaviour_budgets.json in exactly this shape (every clip, every marked letter):

{
  "clips": {
    "clip_01": {
      "A": {
        "timeline": [
          {"start": 0.0, "end": 41.3, "behaviour": "graze"},
          {"start": 41.3, "end": 47.9, "behaviour": "head_up"},
          {"start": 47.9, "end": 52.0, "behaviour": "not_visible"},
          {"start": 52.0, "end": 325.158, "behaviour": "walk"}
        ],
        "budget_seconds": {"walk": 273.2, "trot": 0, "run": 0, "graze": 41.3, "browse": 0, "head_up": 6.6, "auto_groom": 0}
      }
    }
  }
}

The numbers above are placeholders, not a hint about any clip.

  • Write this file early and keep overwriting it. There is a time limit, and whatever is in the file when the session ends is what gets graded. After you finish each animal, rewrite the file with everything you have so far rather than keeping results only in notes or scratch files.
  • Plan your time across all 14 clips and 28 animals; an animal you never get to is worth nothing.

Rules

  • Work only inside this working directory. Do not read, write or search outside it.
  • No internet access is available and none is needed. Do not rely on memory of any dataset, location or recording; every answer must come from what you see in these clips.
  • Tools available are what is installed: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless and matplotlib. There is no network, so pip install will not work.

Media

Frame from clip_01.mp4
clip_01.mp4 · 98 s
Frame from clip_02.mp4
clip_02.mp4 · 94 s
Frame from clip_03.mp4
clip_03.mp4 · 84 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 42 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC0 1.0.

Score

AgentScore (0 to 1)
GPT-6-Astra0.203
Claude Opus 5.50.449
Claude Opus 50.054
Gemini 3.8 Flash0.098
DeepSeek 4.1 Flash0.083

One run per agent.

The agent watches 14 drone clips and writes, for each of 28 marked animals, a timeline of its behaviour and the seconds spent in each behaviour. The reference answer holds 274 bouts.

What a good answer has to get right, and how the verifier checks it

Each marked animal is followed for the whole clip
Each of the 28 animals is scored against its own reference timeline, and the scores are averaged. An animal with no entry scores 0.
The behaviour is named correctly
A reported bout counts only when its behaviour is the same as the reference bout’s, out of seven listed behaviours.
Each change of behaviour is timed
The bout’s start and end must both be within 1.5 s. The tolerance was set from a blind re-coding of 20 reference boundaries (19 could be coded), where the largest error was 1.44 s.
Bouts are neither invented nor missed
The timeline score is an F1 over bouts, so an extra bout and a missed bout both cost. A change lasting 2 to 4 s may be reported or folded into the bouts around it: the answer is scored at floors of 2, 3 and 4 s and the best is kept.
Nothing is claimed about what cannot be seen
Time the reference marks as not visible or as some other behaviour is not checked, and a boundary next to such time may fall anywhere inside it.
The time budget adds up
For each behaviour the total must be within 3 s or 10% of the reference, with more room for behaviours that have many boundaries. More than 3 s reported for a behaviour the animal never shows counts as wrong. The budget is half of each animal’s score.

How the checks become one score

Each animal’s score = 0.5 × timeline F1 + 0.5 × budget score. The budget score is the share of that animal’s behaviour totals that are within tolerance. Score = the mean over all 28 marked animals, each weighing the same whether its clip lasts two minutes or five.

What scores zero

No behaviour_budgets.json or one that cannot be read. An animal that is missing from the answer scores 0 and still counts in the mean, as does an animal with no bout matched and no budget value within tolerance.

Scores from this verifier: the reference answer with one thing changed

SubmissionScoreTimelineBudget
Careful work
The reference answer1.0001.0001.000
Every boundary 0.5 s offEach change of behaviour is reported 0.5 s late or 0.5 s early, alternately, and the time budgets are added up from that shifted timeline.0.9960.9921.000
One kind of mistake
Short bouts left outOnly bouts of 10 s or more are reported; shorter ones are absorbed into the bouts around them.0.5190.3620.677
One look every 10 sThe animal is checked once every 10 s and that behaviour is held until the next check.0.4230.1400.705
Never reports head_upEvery head_up bout is reported as graze. Everything else is correct.0.5640.6000.527
Right timelines, wrong animalsIn clips with several marked animals, each animal is given the correct answer of another marked animal (22 of 28 animals are in such clips).0.2770.2290.324
No real work
One behaviour per animalThe most common behaviour of each animal, held for the whole clip.0.0690.0000.138
No file0.0000.0000.000
An agent’s answer
A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step0.2030.1180.287

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: whatever is reported for time the reference marks as not visible or as some other behaviour, and anything the task does not ask for, such as a written interpretation.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.20
Wall time
21.5 min
Steps shown
106

Loading the run…