AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

Bimanual Hands

hands-bimanual-manipulation-3d-joint-trajectory

You are given three egocentric (head-mounted, first-person) video clips of a person manipulating small tabletop objects with both hands:

Family
Spatial measurement and tracking
Inputs
videodocuments
Media
3 videos, 6 min, no sound track
Reference answer
36 query frames, 720 joint positions
Deliverable
/workspace/output/solution.json
Scoring
geometric (IoU / distance)mean of components

Task package · verifier code

Instruction

Full instruction given to the agent

Right-Hand 3D Joint Trajectory From Egocentric Clips

You are given three egocentric (head-mounted, first-person) video clips of a person manipulating small tabletop objects with both hands:

  • /workspace/materials/clip_01.mp4
  • /workspace/materials/clip_02.mp4
  • /workspace/materials/clip_03.mp4

For a set of query frames in each clip, reconstruct the 3D position of the right hand's 20 skeleton joints, expressed in that clip's RGB camera coordinate frame, in metres.

You are also given, in /workspace/materials/:

  • cameras.json: the pinhole camera intrinsics for each clip, keyed by clip id (clip_01, clip_02, clip_03). Each entry gives model (pinhole), the image width/height, the focal lengths fx/fy, and the principal point cx/cy (standard pinhole model: a 3D camera-frame point [X,Y,Z] projects to pixel u = fx*X/Z + cx, v = fy*Y/Z + cy).
  • queries.json: for each clip id, the list of integer frame indices you must answer for. Frame indices are 0-based into the corresponding mp4.
  • hand_model.json: the bone lengths of the wearer's right hand in metres, constant across the clips (same person throughout). This is your metric reference: with the pinhole intrinsics, a bone of length L at depth Z spans about fx * L / Z pixels, so the observed pixel length of identified bones fixes the metric depth of the hand.

Coordinate frame and joints

  • Camera frame. The origin is the RGB camera optical centre. +Z points forward along the optical axis (into the scene), +X right, +Y down, following the standard pinhole/camera convention that the provided camera model uses. Report every joint as [x, y, z] in metres in this frame, for the clip the query belongs to.
  • The 20 joints, in this exact order (index 0..19):

0 thumb_fingertip 10 index_distal 1 index_fingertip 11 middle_proximal 2 middle_fingertip 12 middle_intermediate 3 ring_fingertip 13 middle_distal 4 pinky_fingertip 14 ring_proximal 5 wrist 15 ring_intermediate 6 thumb_intermediate 16 ring_distal 7 thumb_distal 17 pinky_proximal 8 index_proximal 18 pinky_intermediate 9 index_intermediate 19 pinky_distal

*_proximal / *_intermediate / *_distal are the three finger knuckles from the palm outward; *_fingertip is the tip; wrist is the wrist joint. Report all 20 for every query frame, in the order above.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "clips": {
    "clip_01": [
      {"frame": 137, "joints_m": [[0.031, 0.044, 0.382], "... 20 rows total ..."]},
      {"frame": 402, "joints_m": ["..."]}
    ],
    "clip_02": [{"frame": 88, "joints_m": ["..."]}],
    "clip_03": [{"frame": 51, "joints_m": ["..."]}]
  }
}
  • One object entry per query frame listed in queries.json, under its clip id.
  • frame: the 0-based frame index (must match a frame in queries.json).
  • joints_m: a list of exactly 20 [x, y, z] rows, in the joint order above, in metres, in that clip's camera frame.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online and do not rely on memory; measure the hand from the clips. Use any tools in the image (for example ffmpeg/ffprobe) to seek and sample.
  • Answer only the query frames in queries.json. Answer only the right hand.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.027
Claude Opus 5.50.053
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.003

One run per agent.

The agent gets three first-person clips of a person handling small objects on a table and, for a set of query frames in each clip, gives the 3D position of the 20 joints of the right hand in that clip’s camera coordinates, in metres. The reference holds 36 query frames, 720 joint positions in all.

What a good answer has to get right, and how the verifier checks it

Each joint is placed in metric 3D
For every joint the verifier takes the straight-line distance between the reported and the true position. Depth counts as much as position in the image, so a hand drawn correctly in 2D at the wrong distance scores nothing.
Closer is better, up to 3 cm
Each joint scores 1 − distance / 3 cm: 1.0 for an exact position, 0.5 at 1.5 cm, and 0 at 3 cm or more.
Every query frame is answered
A frame that is missing, malformed or has the wrong number of joints scores 0 for all 20 of its joints and still counts in the mean.

How the checks become one score

Score = the mean joint score over every clip, query frame and joint. All joints weigh the same. A near miss earns part of the credit: there is no single pass-or-fail distance.

What scores zero

No answer or one that cannot be read, or an answer in which every joint is 3 cm or more from its true position.

Outside the score: the left hand, and any frame that is not a query frame.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
tau_m0.03value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.05
Wall time
29.7 min
Steps shown
109

Loading the run…