AgenticVBench-Omni

AgenticVBench-Omni / Asset reconstruction

Blender Expert

blender-expert-session-reimplementation

session.mp4 (1920x1080, 70 min, narrated, with a webcam inset) is one continuous screen recording, cut into chapters, of an expert modelling 3D-printable board-game tokens in Blender 5.1. It starts from a fresh default scene and ends with the expert saving the project. Some stretches show the printed tokens instead of the screen.

Family
Asset reconstruction
Inputs
video
Media
1 video, 70 min, with sound
Reference answer
6 parts, 32 objects
Deliverable
/workspace/output
Scoring
weightedgeometric (IoU / distance)

Task package · verifier code

Instruction

Full instruction given to the agent

Rebuild an expert's Blender project from a screen recording

session.mp4 (1920x1080, 70 min, narrated, with a webcam inset) is one continuous screen recording, cut into chapters, of an expert modelling 3D-printable board-game tokens in Blender 5.1. It starts from a fresh default scene and ends with the expert saving the project. Some stretches show the printed tokens instead of the screen.

Reimplement that project: produce the .blend the expert had at the end of the recording.

Deliverable

output/coin.blend, saved by Blender. Only this file is read.

What must match

The scene is compared with the expert's saved file. Nothing is compared on names of objects, meshes, materials or modifiers, on selection, hide/show state of objects, UI layout or viewport settings.

  1. Scene units: unit system and unit scale.
  2. Collection tree: every collection by exact name (Blender's .001-style suffixes included), its parent collection, and for top-level collections whether it is excluded from the view layer (the outliner checkbox). Each collection must hold the same number of objects of each object type (mesh, curve, text, ...) as the expert's.
  3. Objects, paired with the expert's objects inside the same collection: - parent object; - the modifier stack, in order: modifier type, the viewport and render toggles, and settings. Boolean: operation, operand type, solver, self-intersection and hole-tolerant options, and the operand object. Mirror: axis, bisect and flip toggles, merge toggle and distance, clipping, mirror object. Geometry Nodes modifiers (e.g. the node-based Array from Add Modifier): node group name and every input value. Numbers within 0.001 (absolute) or 0.1 % (relative); - mesh objects: the evaluated geometry in world space (all modifiers applied). Two-way surface distance to the expert's evaluated mesh must be ≤ 0.05 mm on average and ≤ 0.15 mm at the 95th percentile. Vertex counts and topology are not compared; - text objects: body text, font, size (within 1 %), extrude, bevel depth, offset, shear, character/word/line spacing (within 0.02), horizontal and vertical alignment, offset X/Y (within 0.1 mm), text-on-curve target, and world location (within 0.1 mm), rotation (within 0.5°) and scale (within 1 %); - curve objects: world location within 0.1 mm and rotation within 0.5°, spline types, point counts and cyclic flags, and every control point and handle in world space within 0.1 mm.
  4. Look of each printable part: every collection named main or main.NNN is rendered from two fixed oblique cameras (orthographic, flat grey Workbench shading) with only its own objects visible, and must match the expert's renders (SSIM ≥ 0.97 over the part's silhouette).

Each top-level collection counts as one unit and counts only if everything inside it matches.

A few mesh pieces were built off camera (the expert skips ahead). Their geometry is not compared; they must still exist with the right type and collection and be the operands of the right Boolean modifiers, and the part they are joined into is compared as it looks before those joins.

Environment

  • Blender 5.1.2 is installed headless: blender -b ... with --python / --python-expr. There is no display; build the project with Python (bpy) and check it with background renders.
  • The text font the expert used, Arial Bold from the Microsoft core fonts, is at /usr/share/fonts/corefonts/arialbd.ttf (the same path as on the expert's machine).
  • ffmpeg, tesseract and faster-whisper (model small.en, cached) are installed. You cannot hear the audio; transcribe it if you need the narration.

Rules

Work only inside the current working directory. No internet. Do not use any memory you may have of this recording or its author's files; everything needed is in the video.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 1 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 4.0, GPL-3.0-or-later.

Score

AgentScore (0 to 1)
GPT-6-Astra0.156
Claude Opus 5.50.094
Claude Opus 50.094
Gemini 3.8 Flash0.031
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a 70-minute narrated recording of an expert modelling 3D-printable board-game tokens in Blender, from an empty scene to the saved project, and delivers that project as a .blend. The same Blender build reads both files; the verifier compares the structure, the settings, the evaluated geometry and renders. The reference project has 6 parts made of 32 scored objects.

What a good answer has to get right, and how the verifier checks it

The project is organized the same way
The top-level collections, one per printable part plus a helper, must exist with the same parents, the same kinds of object inside and the same child collections, and the scene must use the same units.
Each object is built the same way
The modifier stack and its settings must match, numeric settings within 0.001 in absolute or relative terms, as must text and curve settings.
The geometry is the same
The surface of each object, after all modifiers, is compared in both directions in real units: mean distance at most 0.05 mm and 95th-percentile distance at most 0.15 mm, with location within 0.1 mm and rotation within 0.5°.
The part looks the same
Each part is rendered from fixed views in both files, and the two images must agree with a structural similarity of at least 0.97.
A part is right as a whole
A part counts only when every scored object in it matches and every render of it matches.

How the checks become one score

Score = weight of the parts that pass / total weight, over 6 parts. A part’s weight is its number of scored objects (9, 9, 9, 2, 2 and 1), so a part with many objects counts for more. There is no credit inside a part: one object out of tolerance fails it.

What scores zero

No coin.blend, a file Blender cannot open, or a project in which no part passes in full.

Outside the score: the names of objects, meshes, materials and modifiers, selection and visibility state, and the layout of the interface, which the instruction says are not compared; and 6 objects the specification marks as unscored.

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.16
Wall time
16.9 min
Steps shown
94

Loading the run…