AgenticVBench-Omni

AgenticVBench-Omni / Meetings and procedures

IndustReal

industreal-multivideo-procedure-step-ledger

You are given seven silent egocentric videos at /workspace/materials/A.mp4 through /workspace/materials/G.mp4. Each shows a person assembling or maintaining the same construction-toy car. Reconstruct the complete chronological ledger of every stable assembly-state checkpoint across all seven videos.

Family
Meetings and procedures
Inputs
videodocuments
Media
7 videos, 24 min, no sound track
Reference answer
47 checkpoints
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

State-Transition Ledger Reconstruction

You are given seven silent egocentric videos at /workspace/materials/A.mp4 through /workspace/materials/G.mp4. Each shows a person assembling or maintaining the same construction-toy car. Reconstruct the complete chronological ledger of every stable assembly-state checkpoint across all seven videos.

Use /workspace/materials/overview-of-states.pdf as the visual dictionary for the parts and correct assembly states. Use tools such as ffmpeg and ffprobe to seek, sample, compare, and inspect the videos.

What counts as a checkpoint

Track each component as absent, incorrectly installed, or correctly installed. A checkpoint is anchored to the earliest video frame on which one or more component states have changed. This is the assembly-state transition time, not the end of the surrounding hand action:

  • a correct-install event occurs on the first frame where the component is fully and correctly installed, including any required pin, screw, nut, or tightening;
  • an incorrect-install event occurs on the first frame where the component has been installed in an incorrect state;
  • a removal event occurs on the first frame where a previously correctly or incorrectly installed component is no longer mechanically attached, even if the person is still holding it or sets it down later.

For an incorrect -> absent transition, use that component's normal removal step ID in changes.

If the exact transition is occluded by a hand, use the earliest frame where the new component state can be confirmed. Do not wait for hands to leave the scene, the part to be placed on the desk, the person to inspect the result, or the next action to begin.

The component states visible at the first frame of a video are initial conditions, not checkpoints. For example, do not emit a checkpoint merely because the base or a completed assembly is already present at 00:00.

Report time_s on the absolute timeline of the original A--G video, starting at 0.0 seconds. Do not report a timestamp relative to a seeked subclip or extracted window: if a subclip begins at 80 seconds and the transition is 4 seconds into that subclip, report 84.0, not 4.0.

A single state transition can change multiple components at once. Submit one checkpoint containing every applicable step ID in changes, not separate rows with the same timestamp.

Include all of the following:

  • correct installations;
  • incorrect installations;
  • removals of correctly installed components;
  • removals of incorrectly installed components;
  • repeated removals and reinstallations.

Do not record intermediate hand motions, picking up a loose part, or merely touching or aligning a component before its state changes. Timestamps should be accurate within 2 seconds of the state-transition frame defined above.

State vector

Every checkpoint must include state_after: the complete 11-component state after all changes at that checkpoint. Use -1 for incorrectly installed, 0 for absent, and 1 for correctly installed. The vector always uses this order:

Index Component
0 base
1 front chassis
2 front chassis pin
3 rear chassis
4 short rear chassis
5 front-rear chassis pin
6 rear-rear chassis pin
7 front bracket
8 front bracket screw
9 front wheel assembly
10 rear wheel assembly

Report all 11 entries, including components that did not change at that checkpoint. The values must describe the stable physical state immediately after the transition.

Closed step vocabulary

ID Procedure step
0 Install base
1 Incorrectly install base
2 Remove base
3 Install front chassis
4 Incorrectly install front chassis
5 Remove front chassis
6 Install front chassis pin
7 Incorrectly install front chassis pin
8 Remove front chassis pin
9 Install rear chassis
10 Incorrectly install rear chassis
11 Remove rear chassis
12 Install short rear chassis
13 Incorrectly install short rear chassis
14 Remove short rear chassis
15 Install front-rear chassis pin
16 Incorrectly install front-rear chassis pin
17 Remove front-rear chassis pin
18 Install rear-rear chassis pin
19 Incorrectly install rear-rear chassis pin
20 Remove rear-rear chassis pin
21 Install front bracket
22 Incorrectly install front bracket
23 Remove front bracket
24 Install front bracket screw
25 Incorrectly install front bracket screw
26 Remove front bracket screw
27 Install front wheel assembly
28 Incorrectly install front wheel assembly
29 Remove front wheel assembly
30 Install rear wheel assembly
31 Incorrectly install rear wheel assembly
32 Remove rear wheel assembly

What to submit

Write /workspace/output/solution.json in exactly this shape. The values below are a fictitious format illustration, not an answer row; do not copy them unless supported by the videos:

{
  "checkpoints": [
    {
      "video": "A",
      "time_s": 12.3,
      "changes": [3, 6],
      "state_after": [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
    }
  ]
}
  • video must be one of A through G.
  • time_s is seconds from the start of that video's media timeline.
  • changes contains every applicable integer from the closed vocabulary above, sorted numerically with no duplicates.
  • state_after contains exactly 11 integers in the component order above.
  • Include every checkpoint exactly once. Sort checkpoints by video, then by time.

Rules

  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look up the source recordings, labels, or dataset online.
  • Derive the ledger only from the provided videos and reference PDF.

Media

Media files are distributed separately from the repository. Licences named for this task: see NOTICE.

Score

AgentScore (0 to 1)
GPT-6-Astra0.258
Claude Opus 5.50.430
Claude Opus 50.157
Gemini 3.8 Flash0.178
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets seven silent first-person videos of people assembling or maintaining the same construction-toy car, and a visual dictionary of its parts and correct states. It lists every checkpoint: the video, the earliest moment at which one or more components changed state, which components changed, and the state of all 11 components afterwards. The reference holds 47 checkpoints.

What a good answer has to get right, and how the verifier checks it

Every state change is found, in order
Entries are aligned with the reference in order, and each reference checkpoint can be matched once. A pair can align only in the same video and when the time is within 2 s of the first frame on which the change is visible.
What changed is named
The list of components that changed must be exact.
The full assembly state is right
The state of all 11 components after the change must be exact, including components that are installed incorrectly. An error in an early checkpoint that is carried forward makes the later ones wrong too.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 47 checkpoints and precision over every entry in the file. A checkpoint counts only when video, time, changed components and full state are all right.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no checkpoint is right on every field.

Outside the score: hand movements and partial steps that do not change the state of a component. The verifier also reports matches on time alone and on time and changed components, as diagnostics.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.43
Wall time
12.8 min
Steps shown
397

Loading the run…