You are given three egocentric (head-mounted, first-person) video clips. In each clip a person picks up and turns over one small tabletop object, showing it from many angles:
/workspace/materials/clip_01.mp4
/workspace/materials/clip_02.mp4
/workspace/materials/clip_03.mp4
Full instruction given to the agent
3D Object Reconstruction From Egocentric Manipulation Clips
You are given three egocentric (head-mounted, first-person) video clips. In each clip a
person picks up and turns over one small tabletop object, showing it from many angles:
/workspace/materials/clip_01.mp4
/workspace/materials/clip_02.mp4
/workspace/materials/clip_03.mp4
For each clip, a target object is designated (see objects.json). Reconstruct the
3D surface mesh of each target object from how it appears across that clip.
You are also given, in /workspace/materials/:
objects.json: the target object's name for each clip id (clip_01, clip_02,
clip_03).
cameras.json: the pinhole camera intrinsics for each clip, keyed by clip id, giving
model (pinhole), image width/height, focal lengths fx/fy, and principal
point cx/cy.
What to submit
Write one triangle mesh per clip into /workspace/output/, named by clip id:
/workspace/output/clip_01.obj (or .ply / .glb / .stl)
/workspace/output/clip_02.obj
/workspace/output/clip_03.obj
Each mesh must be a watertight-ish surface reconstruction of that clip's target object:
a set of 3D vertices and triangular faces describing the object's shape. Any of .obj,
.ply, .glb, or .stl is accepted; one file per clip.
Scale and pose do not matter. Your reconstruction is compared to the reference shape
after a best-fit similarity alignment (rotation, translation, and a single global
scale), so you do not need to recover the real-world size or where the object sat, only
its shape. What matters is that the reconstructed surface has the same geometry as
the real object.
Rules
- Stay inside this working directory. Do not read, write, or search outside it.
- Do not look anything up online and do not rely on memory or a stock 3D model of a
similar product; reconstruct the specific object from the clip. Use any tools in the
image (for example
ffmpeg/ffprobe) to seek and sample frames.
- Produce one mesh per clip, for the target object named in
objects.json for that clip.
Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.
One run per agent.
The agent gets three first-person clips, in each of which a person picks up one small object and turns it over, and delivers a 3D triangle mesh of each object. The verifier compares each mesh with a reference mesh of the real object. The reference holds 3 meshes, one object per clip.
What a good answer has to get right, and how the verifier checks it
- Pose and scale are free
- The submitted mesh is first aligned to the reference by the best-fitting rotation, translation and uniform scale, so the agent is not penalized for its choice of units or orientation.
- The surface is in the right place
- 20,000 points are sampled from each mesh. The surface score is an F-score: the share of submitted points that lie near the reference, combined with the share of reference points that lie near the submission. Near means within 2.5 times the typical spacing of the reference points.
- The solid shape is right
- Both meshes are turned into solid volumes on a grid of about 48 cells across, and the overlap of the two volumes is measured as intersection over union. Filling in the hollows of an object, or extruding its outline into a slab, puts surface in the right places but fails here.
How the checks become one score
Each clip’s score = surface F-score squared × volume overlap, each first divided by the value the reference mesh itself reaches when sampled the same way, so that the true shape scores 1. Score = the mean over the three clips. The two terms are multiplied because each alone can be satisfied by a wrong shape.
What scores zero
No mesh files or files that cannot be read as meshes; a missing mesh scores 0 for its clip.
Outside the score: colour and texture, the number of triangles, and the mesh’s position, orientation and scale.
Verifier code
Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.