AgenticVBench-Omni

AgenticVBench-Omni / Scientific observation

Camera Trap

alpine-camera-trap-event-census

You have the full memory-card contents of three motion-triggered camera traps that watched the same small site: a patch of alpine grassland beside a stream, in a mountain national park, during late summer and early autumn (the red-deer rut). The cameras stand about 60 cm above the ground and look at overlapping parts of the site from different angles, so one animal can be on several cameras at the same moment. Every trigger records a video clip with sound (1920x1080, 30 fps). At night the cameras use an infrared flash, so night footage is greyscale. Many triggers are false (wind in the grass, sun, rain) and contain no animal.

Family
Scientific observation
Inputs
videodocuments
Media
356 videos, 1 h 55 min, with sound
Reference answer
55 events
Deliverable
/workspace/output
Scoring
F1 / precision–recallgeometric (IoU / distance)one-to-one matchingJaccardgated product

Task package · verifier code

Instruction

Full instruction given to the agent

Camera-trap event census

You have the full memory-card contents of three motion-triggered camera traps that watched the same small site: a patch of alpine grassland beside a stream, in a mountain national park, during late summer and early autumn (the red-deer rut). The cameras stand about 60 cm above the ground and look at overlapping parts of the site from different angles, so one animal can be on several cameras at the same moment. Every trigger records a video clip with sound (1920x1080, 30 fps). At night the cameras use an infrared flash, so night footage is greyscale. Many triggers are false (wind in the grass, sun, rain) and contain no animal.

Files

  • cameras/A/, cameras/B/, cameras/C/: the clips from each camera, e.g. cameras/B/B0042.mp4.
  • recordings.csv: one row per clip with columns clip (e.g. B/B0042.mp4), camera, clock_start and duration_s. clock_start (YYYY-MM-DD HH:MM:SS) is the moment that camera started recording the clip, as read from that camera's own clock.

There are 356 clips, about 115 minutes in total. The three clocks were set independently and drift; they are not synchronised with each other, the same instant can be stamped up to about half a minute apart on different cameras, and the offsets change over the weeks. Within one camera, the order and spacing of clips are reliable.

What to report

Animals. Only these species count: red_deer, roe_deer, fox, wolf, mountain_hare. Ignore birds, insects, people and anything else.

Events. Events are defined purely from recordings.csv, using the clock values exactly as given (do not correct them for drift). Sort all clips from all three cameras by clock_start. The first clip opens an event. Each next clip joins the current event if its clock_start is at most 300 seconds after the latest end (clock_start + duration_s) of any clip already in that event; otherwise it opens a new event. Every clip belongs to exactly one event. An animal event is an event in which at least one animal appears in at least one of its clips. Report every animal event, and no other events. For each one give:

  • clips: every clip of the event (including clips of that event in which nothing is visible).
  • species: the set of species that appear anywhere in the event.
  • group_size: how many distinct individual animals appear during the event, over all cameras together, as one of "1", "2", "3+". An individual seen on several cameras, or leaving and coming back, counts once.
  • activities: the set of activities (vocabulary below) shown in the event. Measure each activity as the total time, added up over all clips of the event, during which at least one visible animal in that clip shows it (the same moment seen on two cameras counts twice). Every activity with a total of at least 5 s must be listed; an activity shown for less than 5 s in total may be listed or left out; an activity not shown at all must not be listed.
  • first_seen, last_seen: the first and last moment at which any animal of the event is visible on any camera, written as YYYY-MM-DD HH:MM:SS on camera A's clock. A moment seen on camera B or C must be converted to what camera A's clock read at that same instant, even if camera A recorded nothing then.

Activity timeline. For every clip, report the stretches of time during which at least one visible animal is engaged in each activity. If animals do different things at the same time, report overlapping segments with different activities. Report nothing for times when no animal is visible or when you cannot tell what a visible animal is doing. Activities (closed vocabulary):

  • foraging: behaviour aimed at getting food or water, from sensing the environment (e.g. sniffing, searching while walking) to consumption (e.g. grazing, drinking).
  • vigilance: actively sensing the surroundings to detect predators or other threats, or reacting to another animal's call.
  • resting: staying in place for a long time without being vigilant or foraging.
  • grooming: a single animal cleaning its own body or fur, by scratching in any way or while bathing.
  • marking: a single stag marking features of the environment.
  • courtship: breeding behaviour of red deer in the rut, by a single stag (e.g. calling) or involving several animals (e.g. a stag running after a hind).
  • playing: one or more animals, often young, running or jumping with no threat present.
  • chasing: a predator chasing prey.
  • escaping: fleeing a predator, or running away from another animal of the same species.
  • camera_reaction: any behaviour that reacts to the camera itself.

Posture timeline. For every clip, also report the stretches of time during which at least one visible animal is in each posture (closed vocabulary):

  • lying: body resting on the ground.
  • standing_head_up: standing without walking, head held at or above the level of the back.
  • standing_head_down: standing without walking, head lowered below the level of the back (grazing, sniffing, drinking, looking at the ground).
  • walking: moving at a walk.
  • running: moving faster than a walk (trotting, galloping) or jumping.

Moments when any visible animal is doing something that is none of these (scratching, rubbing its antlers, wallowing, shaking its fur) or cannot be made out are not scored for posture.

Vocalizing. A vocalizing span is a continuous stretch of a clip during which an animal visible in that clip is calling (roaring, barking, groaning, mewing, ...) and the call can be heard in that clip's own soundtrack. A pause longer than 0.5 s ends a span. A call heard while no animal is visible in that clip is not a vocalizing span for that clip, even if the caller is on screen on another camera at that moment.

Output

Write output/census.json (names and values below are placeholders):

{
  "events": [
    {"clips": ["A/A0900.mp4", "B/B0901.mp4", "B/B0902.mp4"], "species": ["red_deer"], "group_size": "2",
     "activities": ["foraging", "vigilance"], "first_seen": "2023-09-02 21:14:05", "last_seen": "2023-09-02 21:16:48"}
  ],
  "activity_segments": [
    {"clip": "B/B0901.mp4", "activity": "foraging", "start_s": 0.0, "end_s": 12.4}
  ],
  "posture_segments": [
    {"clip": "B/B0901.mp4", "posture": "standing_head_down", "start_s": 0.0, "end_s": 5.2}
  ],
  "vocalizing_spans": [
    {"clip": "A/A0900.mp4", "start_s": 3.1, "end_s": 7.9}
  ]
}
  • Clip names exactly as in recordings.csv. Times are seconds from the start of that clip's file, 0 <= start_s < end_s <= duration_s.
  • A reported event corresponds to a true event when their clip lists overlap by at least half (intersection over union >= 0.5); its species, group_size and activities must then all be right, and first_seen and last_seen each within 6 s.
  • Activity and posture segments are compared frame by frame; the 0.5 s on either side of each true start or end of an activity or posture is not scored.
  • A vocalizing span counts as found when it overlaps a true span in the same clip with intersection over union >= 0.3; each true span can be found only once.
  • Write the file early and keep it updated as you go. Whatever it contains when the session ends is what gets graded.

Rules

  • Work only inside this directory.
  • There is no internet access. Do not rely on memory of any dataset, site or recording; every answer must come from the files here.
  • Use the tools that are installed: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. pip install will not work.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 357 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY-NC 4.0, mit.

Score

AgentScore (0 to 1)
GPT-6-Astra0.024
Claude Opus 5.50.030
Claude Opus 50.004
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.001

One run per agent.

The agent gets 356 clips from three camera traps whose clocks drift apart, about 115 minutes in all, and writes four things: a list of events (which clips belong together, species, group size, activities, first and last sighting), and frame-by-frame segments of activity, of posture and of vocalizing. The reference answer holds 55 events, 47 of them with an animal.

What a good answer has to get right, and how the verifier checks it

Clips are grouped into the right events
A reported event is paired with a true event when their clip lists overlap by at least half (intersection over union of 0.5 or more), one to one. Clips from different cameras can only be grouped correctly once the three clocks have been reconciled.
Each event is described correctly
A paired event counts only when every field is right: the set of species and the group-size band exactly; every activity shown for 5 s or more listed, and none listed that is not shown at all; first and last sighting each within 6 s on camera A’s clock. An event with no animal in the reference can never count, and every other reported event is a false positive.
Activities are timed
For each of ten activities, the frames where answer and reference both mark it are divided by the frames where either does, pooled over the scored clips. The mean is taken over the activities that have at least 60 s of scored reference.
Postures are timed
The same frame-level measure over five postures. Frames in which a visible animal’s posture is undefined, such as scratching or wallowing, are left out.
Vocalizing is found
A reported span counts when it overlaps a true span in the same clip with intersection over union of 0.3 or more, one to one. The result is an F1 over spans.
Change points are not judged to the frame
The 0.5 s on either side of every true start or end of an activity or posture is not scored, nor are frames whose activity the reference marks as unknown.

How the checks become one score

Score = event F1 × mean activity IoU × mean posture IoU × vocal F1. The four parts are multiplied, not averaged, so the score cannot be higher than the weakest part, and an answer that leaves one part out scores 0.

What scores zero

No answer file or one that cannot be read, or any one of the four parts at zero: no event right on every field, no overlap at all in activity or in posture, or no vocal span found.

Outside the score: the don’t-care clips and frames named above, and reported vocal spans that overlap a region the reference marks as uncertain, which are dropped instead of counted as wrong.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
EVENT_TIME_TOL6constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.02
Wall time
23.1 min
Steps shown
107

Loading the run…