AgenticVBench-Omni

Anonymous artifact for peer review

AgenticVBench-Omni

45 tasks in which an agent turns video, audio and documents into a finished work product, each checked by deterministic code.

Tasks
45
in 8 families
Inputs
43 video
11 also need the audio track; 26 come with documents, scores or maps
Deliverables
37 structured records
7 editable assets or project files, 1 environment state
Scoring
45 verifiers
executable code with ground truth, no model judge
Frame from an input video of Savanna DroneSavanna DroneFrame from an input video of Savanna DroneSavanna DroneFrame from an input video of Driving InfractionsDriving InfractionsFrame from an input video of Driving InfractionsDriving InfractionsFrame from an input video of Siren YieldSiren YieldFrame from an input video of Siren YieldSiren YieldFrame from an input video of Hidden SoundHidden SoundFrame from an input video of Hidden SoundHidden SoundFrame from an input video of House WalkthroughHouse WalkthroughFrame from an input video of House WalkthroughHouse WalkthroughFrame from an input video of Object WhereaboutsObject WhereaboutsFrame from an input video of Object WhereaboutsObject WhereaboutsFrame from an input video of A/V SyncA/V SyncFrame from an input video of A/V SyncA/V SyncFrame from an input video of Piano DuoPiano DuoFrame from an input video of Piano DuoPiano Duo

Frames from the input videos of 8 tasks whose media is rendered for the benchmark or released under CC0. Scroll sideways for more.

All 45 tasks

Filter by family or search by name. Each task page gives the instruction, the input media, the scores of the five evaluated agents with how the deliverable is scored, and a sampled run step by step.

Sports analysis8 tasks

Badmintonbadminton-momota-chou-2019-long-rally-checkpointsvideodocuments1 video, 1 h 43 min, with sound
Reference answer: 26 rallies and 78 checkpoints
F1 / precision–recall · gated product
Balance Beampac12-2018-balance-beam-routine-timelinevideodocuments1 video, 2 h 19 min, with sound
Reference answer: 23 routines
F1 / precision–recall · one-to-one matching
start tolerance 0.25 s, end tolerance 0.25 s, dismount takeoff tolerance 0.25 s
BYU Volleyballbyu-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 55 min, no sound track
Reference answer: 18 block points
F1 / precision–recall · partial credit
Chessotb-chess-move-capture-reconstructionvideo1 video, 26 min, no sound track
Reference answer: 104 moves and 26 captures
checklist / accuracy
time tolerance 6 s, ply match window 3
GSWgsw-cle-2018-finals-g4-three-point-timelinevideo1 video, 2 h 35 min, with sound
Reference answer: 22 made three-pointers
F1 / precision–recall · partial credit
clock tolerance 5 s
Lacrosselacrosse-mich-jhu-2024videodocuments1 video, 1 h 42 min, no sound track
Reference answer: 26 goals
partial credit · F1 / precision–recall
OpenTTGamesopenttgames-rally-event-chainvideo1 video, 24 min, no sound track
Reference answer: 92 rallies and 387 strokes
F1 / precision–recall · gated product
rally start tolerance 1 s, stroke tolerance 0.35 s, ending tolerance 1 s
USC Volleyballusc-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 58 min, no sound track
Reference answer: 23 block points
F1 / precision–recall · partial credit

Game-state reconstruction6 tasks

Scientific observation6 tasks

Transport and operations4 tasks

Meetings and procedures5 tasks

Spatial measurement and tracking6 tasks

Music and synchronization3 tasks

Asset reconstruction7 tasks