AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

MEVA

meva-target-person-activity-ledgers

You are given:

Family
Spatial measurement and tracking
Inputs
videodocuments
Media
1 video, 10 min, no sound track
Reference answer
29 activity occurrences
Deliverable
/workspace/output/solution.json
Scoring
soft F1weightedF1 / precision–recallgated product

Task package · verifier code

Instruction

Full instruction given to the agent

Target-Person Activity Ledger Reconstruction

You are given:

  • /workspace/materials/meva_activity_montage.mp4: one silent surveillance montage.
  • /workspace/materials/roster.json: ten anonymous target references.
  • /workspace/materials/roster/reference_001.jpg through /workspace/materials/roster/reference_010.jpg: one image per target.

Reconstruct every visible occurrence of the allowed activities performed by each pictured target. The references identify video-local targets only; do not infer that people from different scenes or cameras are the same physical person.

Allowed activity types

  • person_picks_up_object: the target lifts an object from a supporting surface or the ground.
  • person_carries_heavy_object: the target visibly transports a bulky or heavy object.
  • person_puts_down_object: the target places a carried object onto a supporting surface or the ground.
  • person_opens_vehicle_door: the target opens a vehicle door.
  • person_enters_vehicle: the target moves from outside to inside a vehicle.
  • person_closes_vehicle_door: the target closes a vehicle door.

Report overlapping activities separately. Use seconds from the beginning of the montage. A temporal match allows up to 3.0 seconds of midpoint error, but start and end times should describe the visible activity interval as accurately as possible.

Output

Write /workspace/output/solution.json:

{
  "ledgers": [
    {
      "reference_id": "reference_001",
      "events": [
        {
          "activity_type": "person_picks_up_object",
          "start_time_s": 12.3,
          "end_time_s": 13.1
        }
      ]
    }
  ]
}

Requirements:

  • Use only reference_001 through reference_010.
  • Include each target once. Use an empty events list when no qualifying event is found.
  • List each target's events chronologically.
  • Do not duplicate an occurrence.
  • Inspect the complete montage.
  • Do not use online lookup or external knowledge.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.033
Claude Opus 5.50.024
Claude Opus 50.064
Gemini 3.8 Flash0.008
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets a 10-minute silent montage of surveillance footage and pictures of ten people. For each pictured person it lists every occurrence of six activities, such as picking up an object or entering a vehicle, with start and end times. The reference holds 29 activity occurrences.

What a good answer has to get right, and how the verifier checks it

The activity is given to the right person
A listed occurrence can only be paired with a true occurrence of the same person and the same activity type.
The occurrence is placed in time
The midpoint of the listed interval must be within 3 s of the true midpoint; beyond that the pair scores 0.
Closer timing earns more
A pair scores 0.5 × (1 − midpoint error / 3 s) + 0.3 × interval overlap (intersection over union) + 0.2 × the ratio of the shorter duration to the longer. An exact interval scores 1.
Occurrences are neither invented nor missed
Within each person and activity, pairs are chosen in time order to give the largest total. The total is turned into two F1 values: one over all occurrences, 2 × total / (listed + 29), and one computed per person and averaged over the ten people.

How the checks become one score

Score = overall F1 × per-person mean F1. Multiplying the two means that doing well on the people with many occurrences cannot make up for getting other people wrong. A person with no true occurrence and none listed counts as 1 in the per-person mean.

What scores zero

No solution file or one over 1 MB, or a file with any structural error: an unknown or repeated person ID, an unknown activity type, a non-numeric time, or an interval that is empty or runs past the 600 s of the montage. One such entry voids the whole file. Otherwise, an answer in which no occurrence is within 3 s.

Outside the score: activities other than the six listed, and people who are not among the ten pictured.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
MIDPOINT_TOLERANCE_S3constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5 · Claude Code 2.1.269
Score
0.06
Wall time
49.4 min
Steps shown
155

Loading the run…