AgenticVBench-Omni

AgenticVBench-Omni / Asset reconstruction

Object Reconstruction

object-reconstruction-from-egocentric-manipulation

You are given three egocentric (head-mounted, first-person) video clips. In each clip a person picks up and turns over one small tabletop object, showing it from many angles:

Family
Asset reconstruction
Inputs
videodocuments
Media
3 videos, 6 min, no sound track
Reference answer
3 object meshes
Deliverable
/workspace/output
Scoring
F1 / precision–recallper-componentgeometric (IoU / distance)mean of components

Task package · verifier code

Instruction

Full instruction given to the agent

3D Object Reconstruction From Egocentric Manipulation Clips

You are given three egocentric (head-mounted, first-person) video clips. In each clip a person picks up and turns over one small tabletop object, showing it from many angles:

  • /workspace/materials/clip_01.mp4
  • /workspace/materials/clip_02.mp4
  • /workspace/materials/clip_03.mp4

For each clip, a target object is designated (see objects.json). Reconstruct the 3D surface mesh of each target object from how it appears across that clip.

You are also given, in /workspace/materials/:

  • objects.json: the target object's name for each clip id (clip_01, clip_02, clip_03).
  • cameras.json: the pinhole camera intrinsics for each clip, keyed by clip id, giving model (pinhole), image width/height, focal lengths fx/fy, and principal point cx/cy.

What to submit

Write one triangle mesh per clip into /workspace/output/, named by clip id:

  • /workspace/output/clip_01.obj (or .ply / .glb / .stl)
  • /workspace/output/clip_02.obj
  • /workspace/output/clip_03.obj

Each mesh must be a watertight-ish surface reconstruction of that clip's target object: a set of 3D vertices and triangular faces describing the object's shape. Any of .obj, .ply, .glb, or .stl is accepted; one file per clip.

Scale and pose do not matter. Your reconstruction is compared to the reference shape after a best-fit similarity alignment (rotation, translation, and a single global scale), so you do not need to recover the real-world size or where the object sat, only its shape. What matters is that the reconstructed surface has the same geometry as the real object.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online and do not rely on memory or a stock 3D model of a similar product; reconstruct the specific object from the clip. Use any tools in the image (for example ffmpeg/ffprobe) to seek and sample frames.
  • Produce one mesh per clip, for the target object named in objects.json for that clip.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.043
Claude Opus 5.50.030
Claude Opus 50.024
Gemini 3.8 Flash0.114
DeepSeek 4.1 Flash0.003

One run per agent.

The agent gets three first-person clips, in each of which a person picks up one small object and turns it over, and delivers a 3D triangle mesh of each object. The verifier compares each mesh with a reference mesh of the real object. The reference holds 3 meshes, one object per clip.

What a good answer has to get right, and how the verifier checks it

Pose and scale are free
The submitted mesh is first aligned to the reference by the best-fitting rotation, translation and uniform scale, so the agent is not penalized for its choice of units or orientation.
The surface is in the right place
20,000 points are sampled from each mesh. The surface score is an F-score: the share of submitted points that lie near the reference, combined with the share of reference points that lie near the submission. Near means within 2.5 times the typical spacing of the reference points.
The solid shape is right
Both meshes are turned into solid volumes on a grid of about 48 cells across, and the overlap of the two volumes is measured as intersection over union. Filling in the hollows of an object, or extruding its outline into a slab, puts surface in the right places but fails here.

How the checks become one score

Each clip’s score = surface F-score squared × volume overlap, each first divided by the value the reference mesh itself reaches when sampled the same way, so that the true shape scores 1. Score = the mean over the three clips. The two terms are multiplied because each alone can be satisfied by a wrong shape.

What scores zero

No mesh files or files that cannot be read as meshes; a missing mesh scores 0 for its clip.

Outside the score: colour and texture, the number of triangles, and the mesh’s position, orientation and scale.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
per_clip.clip_01.tau_m0.0018value the verifier reports using
per_clip.clip_02.tau_m0.00156value the verifier reports using
per_clip.clip_03.tau_m0.00143value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Gemini 3.8 Flash · Antigravity CLI 1.2.2
Score
0.11
Wall time
10.2 min
Steps shown
147

Loading the run…