AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

Tabletop Objects

objects-tabletop-manipulation-6dof-pose-trajectory

You are given three egocentric (head-mounted, first-person) video clips of a person manipulating small tabletop objects:

Family
Spatial measurement and tracking
Inputs
videodocuments
Media
3 videos, 6 min, no sound track
Reference answer
36 poses
Deliverable
/workspace/output/solution.json
Scoring
per-componentgeometric (IoU / distance)weightedmean of components

Task package · verifier code

Instruction

Full instruction given to the agent

Object 6DoF Pose Trajectory From Egocentric Clips

You are given three egocentric (head-mounted, first-person) video clips of a person manipulating small tabletop objects:

  • /workspace/materials/clip_01.mp4
  • /workspace/materials/clip_02.mp4
  • /workspace/materials/clip_03.mp4

In each clip one target object is designated. For a set of query frames in each clip, reconstruct the 6DoF pose of that target object (its position and orientation), expressed in that clip's RGB camera coordinate frame, in metres.

You are also given, in /workspace/materials/:

  • objects.json: the target object's name for each clip id (clip_01, clip_02, clip_03).
  • cameras.json: the pinhole camera intrinsics for each clip, keyed by clip id, giving model (pinhole), image width/height, focal lengths fx/fy, and principal point cx/cy (a 3D camera-frame point [X,Y,Z] projects to pixel u = fx*X/Z + cx, v = fy*Y/Z + cy).
  • queries.json: for each clip id, the list of 0-based integer frame indices you must answer for.
  • object_points.json: for each clip id, a set of 3D points sampled from the target object's surface, given in the object's own canonical frame (metres). This defines the canonical frame your pose maps from, and is the same point set the pose is scored on.

What a pose means here

The pose you report for a query frame is the rigid transform T_camera_object that maps a point p_obj given in the object's canonical frame (the frame object_points.json is in) to the camera frame:

p_camera = R * p_obj + t
  • t = [tx, ty, tz] is the object origin in the camera frame, in metres (camera convention: +Z forward along the optical axis, +X right, +Y down).
  • R is the rotation, which you report as a unit quaternion q = [w, x, y, z].

Recover a pose per query frame such that the object points, placed by R and t and projected through the given camera model, line up with the object as it appears in that frame. Because you have both the object points and the camera model, a correct pose is well defined; the difficulty is estimating it from a single moving view.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "clips": {
    "clip_01": [
      {"frame": 210, "t_xyz_m": [0.05, -0.02, 0.44], "q_wxyz": [0.98, 0.01, -0.17, 0.02]},
      {"frame": 655, "t_xyz_m": ["..."], "q_wxyz": ["..."]}
    ],
    "clip_02": [{"frame": 88, "t_xyz_m": ["..."], "q_wxyz": ["..."]}],
    "clip_03": [{"frame": 51, "t_xyz_m": ["..."], "q_wxyz": ["..."]}]
  }
}
  • One entry per query frame listed in queries.json, under its clip id.
  • frame: the 0-based frame index (must match a frame in queries.json).
  • t_xyz_m: object origin in the camera frame, metres.
  • q_wxyz: unit quaternion (w first) for the object's orientation in the camera frame.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online and do not rely on memory; measure the object from the clips. Use any tools in the image (for example ffmpeg/ffprobe) to seek and sample.
  • Answer only the query frames in queries.json, and only the target object named in objects.json for that clip.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.329
Claude Opus 5.50.325
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets three first-person clips of a person handling small objects on a table. In each clip one object is designated, and for a set of query frames the agent gives its full pose in that clip’s camera coordinates: position in metres and orientation as a quaternion. The reference holds 36 query frames.

What a good answer has to get right, and how the verifier checks it

Position and orientation are both right
The verifier moves the object’s reference point set by the reported pose and by the true pose and averages the distance between corresponding points (the ADD measure used in pose-estimation benchmarks). A wrong rotation and a wrong depth both show up as distance.
Closer is better, up to a tenth of the object’s size
Each frame scores 1 − average distance / (10% of the object’s diameter): 1.0 for an exact pose, 0 once the average distance reaches a tenth of the diameter.
Symmetry is allowed for
For an object flagged as symmetric each point is compared with the nearest true point instead, so a rotation that leaves the object looking the same is not penalized. None of the three objects here is flagged.
Every query frame is answered
A missing or malformed frame scores 0 and still counts in the mean.

How the checks become one score

Score = the mean frame score over all query frames of the three clips. A near miss earns part of the credit: there is no single pass-or-fail distance.

What scores zero

No answer or one that cannot be read, or an answer in which every pose is off by a tenth of the object’s diameter or more.

Outside the score: any frame that is not a query frame, and the other objects on the table.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
per_clip.clip_01.tau_m0.03454value the verifier reports using
per_clip.clip_02.tau_m0.02375value the verifier reports using
per_clip.clip_03.tau_m0.03103value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.33
Wall time
12.5 min
Steps shown
53

Loading the run…