Anonymous artifact for peer review
AgenticVBench-Omni
45 tasks in which an agent turns video, audio and documents into a finished work product, each checked by deterministic code.
Savanna Drone
Savanna Drone
Driving Infractions
Driving Infractions
Siren Yield
Siren Yield
Hidden Sound
Hidden Sound
House Walkthrough
House Walkthrough
Object Whereabouts
Object Whereabouts
A/V Sync
A/V Sync
Piano Duo
Piano DuoFrames from the input videos of 8 tasks whose media is rendered for the benchmark or released under CC0. Scroll sideways for more.
All 45 tasks
Filter by family or search by name. Each task page gives the instruction, the input media, the scores of the five evaluated agents with how the deliverable is scored, and a sampled run step by step.
Sports analysis8 tasks
Badmintonbadminton-momota-chou-2019-long-rally-checkpointsvideodocuments1 video, 1 h 43 min, with sound
Reference answer: 26 rallies and 78 checkpointsF1 / precision–recall · gated productBalance Beampac12-2018-balance-beam-routine-timelinevideodocuments1 video, 2 h 19 min, with sound
Reference answer: 23 routinesF1 / precision–recall · one-to-one matching
start tolerance 0.25 s, end tolerance 0.25 s, dismount takeoff tolerance 0.25 sBYU Volleyballbyu-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 55 min, no sound track
Reference answer: 18 block pointsF1 / precision–recall · partial creditChessotb-chess-move-capture-reconstructionvideo1 video, 26 min, no sound track
Reference answer: 104 moves and 26 captureschecklist / accuracy
time tolerance 6 s, ply match window 3GSWgsw-cle-2018-finals-g4-three-point-timelinevideo1 video, 2 h 35 min, with sound
Reference answer: 22 made three-pointersF1 / precision–recall · partial credit
clock tolerance 5 sLacrosselacrosse-mich-jhu-2024videodocuments1 video, 1 h 42 min, no sound track
Reference answer: 26 goalspartial credit · F1 / precision–recallOpenTTGamesopenttgames-rally-event-chainvideo1 video, 24 min, no sound track
Reference answer: 92 rallies and 387 strokesF1 / precision–recall · gated product
rally start tolerance 1 s, stroke tolerance 0.35 s, ending tolerance 1 sUSC Volleyballusc-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 58 min, no sound track
Reference answer: 23 block pointsF1 / precision–recall · partial credit
Reference answer: 26 rallies and 78 checkpointsF1 / precision–recall · gated productBalance Beampac12-2018-balance-beam-routine-timelinevideodocuments1 video, 2 h 19 min, with sound
Reference answer: 23 routinesF1 / precision–recall · one-to-one matching
start tolerance 0.25 s, end tolerance 0.25 s, dismount takeoff tolerance 0.25 sBYU Volleyballbyu-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 55 min, no sound track
Reference answer: 18 block pointsF1 / precision–recall · partial creditChessotb-chess-move-capture-reconstructionvideo1 video, 26 min, no sound track
Reference answer: 104 moves and 26 captureschecklist / accuracy
time tolerance 6 s, ply match window 3GSWgsw-cle-2018-finals-g4-three-point-timelinevideo1 video, 2 h 35 min, with sound
Reference answer: 22 made three-pointersF1 / precision–recall · partial credit
clock tolerance 5 sLacrosselacrosse-mich-jhu-2024videodocuments1 video, 1 h 42 min, no sound track
Reference answer: 26 goalspartial credit · F1 / precision–recallOpenTTGamesopenttgames-rally-event-chainvideo1 video, 24 min, no sound track
Reference answer: 92 rallies and 387 strokesF1 / precision–recall · gated product
rally start tolerance 1 s, stroke tolerance 0.35 s, ending tolerance 1 sUSC Volleyballusc-wsu-2023-volleyball-block-timelinevideo1 video, 1 h 58 min, no sound track
Reference answer: 23 block pointsF1 / precision–recall · partial credit
Game-state reconstruction6 tasks
DOOMdoom-checkpoint-state-trackingvideo1 video, 44 min, no sound track
Reference answer: 138 eventsper-component · checklist / accuracy
timestamp tolerance 750 msDota 2dota2-pgl-wallachia-s7-game5-predeath-trajectoriesvideo1 video, 52 min, no sound track
Reference answer: 39 deathsF1 / precision–recall · geometric (IoU / distance)
clock tolerance 2 sLoLlol-minimap-timeline-reconstructionvideo1 video, 40 min, no sound track
Reference answer: 111 eventsF1 / precision–recall · one-to-one matching
tolerance 3 sMeleemelee-nouns-bowl-2025-combo-kill-causal-ledgervideo1 video, 25 min, no sound track
Reference answer: 104 conversionsF1 / precision–recallMinecraftminecraft-gameplay-ledger-s1video1 video, 3 h 58 min, no sound track
Reference answer: 1,995 eventsF1 / precision–recall · weighted
time tolerance 10StarCraft IIsc2-build-order-reconstructionvideodocuments9 videos, 50 min, no sound track
Reference answer: 94 build eventsF1 / precision–recall · one-to-one matching
tolerance 3 s
Reference answer: 138 eventsper-component · checklist / accuracy
timestamp tolerance 750 msDota 2dota2-pgl-wallachia-s7-game5-predeath-trajectoriesvideo1 video, 52 min, no sound track
Reference answer: 39 deathsF1 / precision–recall · geometric (IoU / distance)
clock tolerance 2 sLoLlol-minimap-timeline-reconstructionvideo1 video, 40 min, no sound track
Reference answer: 111 eventsF1 / precision–recall · one-to-one matching
tolerance 3 sMeleemelee-nouns-bowl-2025-combo-kill-causal-ledgervideo1 video, 25 min, no sound track
Reference answer: 104 conversionsF1 / precision–recallMinecraftminecraft-gameplay-ledger-s1video1 video, 3 h 58 min, no sound track
Reference answer: 1,995 eventsF1 / precision–recall · weighted
time tolerance 10StarCraft IIsc2-build-order-reconstructionvideodocuments9 videos, 50 min, no sound track
Reference answer: 94 build eventsF1 / precision–recall · one-to-one matching
tolerance 3 s
Scientific observation6 tasks
Aviaryaviary-caller-attribution-ledgervideoaudiodocuments21 videos, 21 min, no sound track; 21 audio files, 21 min
Reference answer: 143 scored boutsF1 / precision–recall · one-to-one matching
tolerance 0.1 sCamera Trapalpine-camera-trap-event-censusvideodocuments356 videos, 1 h 55 min, with sound
Reference answer: 55 eventsF1 / precision–recall · geometric (IoU / distance)
event time tolerance 6Cell lineagecell-division-lineage-censusvideo1 video, 4 min, no sound track
Reference answer: 257 divisionsF1 / precision–recall · one-to-one matching
frame tolerance 5, pos tolerance 25Pigeon Flockpigeon-flock-mocap-ledgervideodocuments24 videos, 4 h 8 min, no sound track
Reference answer: 3,481 positions and 32 displacementsF1 / precision–recall · geometric (IoU / distance)
tolerance 0.12 m, tolerance 1.5 sSavanna Dronesavanna-drone-behaviour-budgetsvideodocuments14 videos, 49 min, no sound track
Reference answer: 274 bouts of 28 animalsF1 / precision–recall · mean of components
1.5 s, 3.0 sSiberian Jaysiberian-jay-feeder-individual-ledgervideodocuments4 videos, 1 h 45 min, no sound track
Reference answer: 104 boutsF1 / precision–recall
0.3 seconds, 2 seconds
Reference answer: 143 scored boutsF1 / precision–recall · one-to-one matching
tolerance 0.1 sCamera Trapalpine-camera-trap-event-censusvideodocuments356 videos, 1 h 55 min, with sound
Reference answer: 55 eventsF1 / precision–recall · geometric (IoU / distance)
event time tolerance 6Cell lineagecell-division-lineage-censusvideo1 video, 4 min, no sound track
Reference answer: 257 divisionsF1 / precision–recall · one-to-one matching
frame tolerance 5, pos tolerance 25Pigeon Flockpigeon-flock-mocap-ledgervideodocuments24 videos, 4 h 8 min, no sound track
Reference answer: 3,481 positions and 32 displacementsF1 / precision–recall · geometric (IoU / distance)
tolerance 0.12 m, tolerance 1.5 sSavanna Dronesavanna-drone-behaviour-budgetsvideodocuments14 videos, 49 min, no sound track
Reference answer: 274 bouts of 28 animalsF1 / precision–recall · mean of components
1.5 s, 3.0 sSiberian Jaysiberian-jay-feeder-individual-ledgervideodocuments4 videos, 1 h 45 min, no sound track
Reference answer: 104 boutsF1 / precision–recall
0.3 seconds, 2 seconds
Transport and operations4 tasks
Driving Infractionscarla-surround-infraction-ledgervideodocuments8 videos, 16 h, no sound track
Reference answer: 136 infractionsF1 / precision–recall
time tolerance 0.6 sDual Runwaydualrunway-ops-ledgervideoaudio1 video, 2 h 34 min, no sound track; 1 audio file, 2 h 36 min
Reference answer: 49 operationsF1 / precision–recall
dont care tolerance 120Siren Yieldcarla-siren-yield-ledgervideoaudiodocuments4 videos, 6 h, no sound track; 1 audio file, 1 h 30 min
Reference answer: 31 vehiclesF1 / precision–recall · one-to-one matching
0.5 s, 1.0 sYear of Drivingyear-of-driving-route-ledgervideodocuments82 videos, 4 h 6 min, no sound track
Reference answer: 205 street segments and 63 place passesF1 / precision–recall · Jaccard
15 s, 15 seconds, 40 m
Reference answer: 136 infractionsF1 / precision–recall
time tolerance 0.6 sDual Runwaydualrunway-ops-ledgervideoaudio1 video, 2 h 34 min, no sound track; 1 audio file, 2 h 36 min
Reference answer: 49 operationsF1 / precision–recall
dont care tolerance 120Siren Yieldcarla-siren-yield-ledgervideoaudiodocuments4 videos, 6 h, no sound track; 1 audio file, 1 h 30 min
Reference answer: 31 vehiclesF1 / precision–recall · one-to-one matching
0.5 s, 1.0 sYear of Drivingyear-of-driving-route-ledgervideodocuments82 videos, 4 h 6 min, no sound track
Reference answer: 205 street segments and 63 place passesF1 / precision–recall · Jaccard
15 s, 15 seconds, 40 m
Meetings and procedures5 tasks
Cookingcaptaincook-recipe-step-transcriptsvideo22 videos, 4 h 53 min, no sound track
Reference answer: 314 stepsF1 / precision–recall · partial creditCouncilcouncil-meeting-motion-ledgervideoaudiodocuments14 videos, 52 h 33 min, no sound track; 14 audio files, 52 h 40 min
Reference answer: 103 motionsF1 / precision–recall
2 sEuroparleuroparl-2026-single-market-speaker-turn-ledgervideoaudiodocuments1 video, 2 h 8 min, with sound
Reference answer: 86 speaker turnsF1 / precision–recall
boundary tolerance 3.5 sIndustRealindustreal-multivideo-procedure-step-ledgervideodocuments7 videos, 24 min, no sound track
Reference answer: 47 checkpointsF1 / precision–recall
2 secondsRobot Recoveryrobot-fault-recoveryinteractive simulatora live robot simulator with two cameras; no media files
Reference answer: 8 checklist stepsper-component · checklist / accuracy
rest tolerance 0.1, pos tolerance 0.05, vert tolerance 0.08
Reference answer: 314 stepsF1 / precision–recall · partial creditCouncilcouncil-meeting-motion-ledgervideoaudiodocuments14 videos, 52 h 33 min, no sound track; 14 audio files, 52 h 40 min
Reference answer: 103 motionsF1 / precision–recall
2 sEuroparleuroparl-2026-single-market-speaker-turn-ledgervideoaudiodocuments1 video, 2 h 8 min, with sound
Reference answer: 86 speaker turnsF1 / precision–recall
boundary tolerance 3.5 sIndustRealindustreal-multivideo-procedure-step-ledgervideodocuments7 videos, 24 min, no sound track
Reference answer: 47 checkpointsF1 / precision–recall
2 secondsRobot Recoveryrobot-fault-recoveryinteractive simulatora live robot simulator with two cameras; no media files
Reference answer: 8 checklist stepsper-component · checklist / accuracy
rest tolerance 0.1, pos tolerance 0.05, vert tolerance 0.08
Spatial measurement and tracking6 tasks
Bimanual Handshands-bimanual-manipulation-3d-joint-trajectoryvideodocuments3 videos, 6 min, no sound track
Reference answer: 36 query frames, 720 joint positionsgeometric (IoU / distance) · mean of components
tau 0.03 mHidden Soundhidden-sound-source-mapvideoaudiodocuments1 video, 30 min, with sound
Reference answer: 24 sources with 98 on-intervalsF1 / precision–recall · one-to-one matching
pos tolerance 1 m, time tolerance 5 sHouse Walkthroughhouse-walkthrough-floorplan-inventoryvideodocuments1 video, 54 min, no sound track
Reference answer: 10 rooms, 10 doorways and 125 objectsF1 / precision–recall · one-to-one matching
0.25 mMEVAmeva-target-person-activity-ledgersvideodocuments1 video, 10 min, no sound track
Reference answer: 29 activity occurrencessoft F1 · weighted
midpoint tolerance 3 sObject Whereaboutsobject-whereabouts-day-ledgervideodocuments1 video, 2 h 50 min, no sound track
Reference answer: 302 scored answers for 57 objectschecklist / accuracyTabletop Objectsobjects-tabletop-manipulation-6dof-pose-trajectoryvideodocuments3 videos, 6 min, no sound track
Reference answer: 36 posesper-component · geometric (IoU / distance)
tau 0.03454 m, tau 0.02375 m, tau 0.03103 m
Reference answer: 36 query frames, 720 joint positionsgeometric (IoU / distance) · mean of components
tau 0.03 mHidden Soundhidden-sound-source-mapvideoaudiodocuments1 video, 30 min, with sound
Reference answer: 24 sources with 98 on-intervalsF1 / precision–recall · one-to-one matching
pos tolerance 1 m, time tolerance 5 sHouse Walkthroughhouse-walkthrough-floorplan-inventoryvideodocuments1 video, 54 min, no sound track
Reference answer: 10 rooms, 10 doorways and 125 objectsF1 / precision–recall · one-to-one matching
0.25 mMEVAmeva-target-person-activity-ledgersvideodocuments1 video, 10 min, no sound track
Reference answer: 29 activity occurrencessoft F1 · weighted
midpoint tolerance 3 sObject Whereaboutsobject-whereabouts-day-ledgervideodocuments1 video, 2 h 50 min, no sound track
Reference answer: 302 scored answers for 57 objectschecklist / accuracyTabletop Objectsobjects-tabletop-manipulation-6dof-pose-trajectoryvideodocuments3 videos, 6 min, no sound track
Reference answer: 36 posesper-component · geometric (IoU / distance)
tau 0.03454 m, tau 0.02375 m, tau 0.03103 m
Music and synchronization3 tasks
A/V Syncav-sync-drift-ledgervideoaudiodocuments1 video, 4 h 21 min, with sound
Reference answer: 35 faulty stretchesF1 / precision–recall · one-to-one matching
1 s, 1.0 s, 40 msPiano Duopiano-duo-score-deviation-ledgervideoaudiodocuments1 video, 25 min, with sound
Reference answer: 186 departuresF1 / precision–recall · one-to-one matching
beat tolerance 0.125Symphonysymphony-wrong-note-ledgeraudiodocuments9 audio files, 2 h 2 min
Reference answer: 399 departuresF1 / precision–recall · one-to-one matching
beat tolerance 0.25, extra beat tolerance 0.5
Reference answer: 35 faulty stretchesF1 / precision–recall · one-to-one matching
1 s, 1.0 s, 40 msPiano Duopiano-duo-score-deviation-ledgervideoaudiodocuments1 video, 25 min, with sound
Reference answer: 186 departuresF1 / precision–recall · one-to-one matching
beat tolerance 0.125Symphonysymphony-wrong-note-ledgeraudiodocuments9 audio files, 2 h 2 min
Reference answer: 399 departuresF1 / precision–recall · one-to-one matching
beat tolerance 0.25, extra beat tolerance 0.5
Asset reconstruction7 tasks
Ardourardour-mixing-session-reimplementationvideoaudio1 video, 15 min, with sound; 21 audio files, 6 min
Reference answer: 134 scored itemsper-component · checklist / accuracy
0.5 s, 3 msBlender Advancedblender-advanced-session-reimplementationvideo1 video, 42 min, with sound
Reference answer: 23 node trees with 838 nodesper-componentBlender Expertblender-expert-session-reimplementationvideo1 video, 70 min, with sound
Reference answer: 6 parts, 32 objectsweighted · geometric (IoU / distance)
0.1 mmClassroomabandoned-classroom-scene-reimplementationvideodocuments1 video, 72 min, with sound
Reference answer: 47 desks, 18 chairs and 41 materialsper-componentObject Reconstructionobject-reconstruction-from-egocentric-manipulationvideodocuments3 videos, 6 min, no sound track
Reference answer: 3 object meshesF1 / precision–recall · per-component
tau 0.0018 m, tau 0.00156 m, tau 0.00143 mSlide Deckslide-deck-session-reimplementationvideo1 video, 14 min, with sound
Reference answer: 29 slides with 536 shapesper-component · one-to-one matching
0.06 inch, 0.12 inchSynthesizeraudio-production-session-reimplementationvideoaudio1 video, 2 h 34 min, with sound
Reference answer: 9 synth patchesper-component · mean of components
float tolerance 0.025, seq tolerance 0.02
Reference answer: 134 scored itemsper-component · checklist / accuracy
0.5 s, 3 msBlender Advancedblender-advanced-session-reimplementationvideo1 video, 42 min, with sound
Reference answer: 23 node trees with 838 nodesper-componentBlender Expertblender-expert-session-reimplementationvideo1 video, 70 min, with sound
Reference answer: 6 parts, 32 objectsweighted · geometric (IoU / distance)
0.1 mmClassroomabandoned-classroom-scene-reimplementationvideodocuments1 video, 72 min, with sound
Reference answer: 47 desks, 18 chairs and 41 materialsper-componentObject Reconstructionobject-reconstruction-from-egocentric-manipulationvideodocuments3 videos, 6 min, no sound track
Reference answer: 3 object meshesF1 / precision–recall · per-component
tau 0.0018 m, tau 0.00156 m, tau 0.00143 mSlide Deckslide-deck-session-reimplementationvideo1 video, 14 min, with sound
Reference answer: 29 slides with 536 shapesper-component · one-to-one matching
0.06 inch, 0.12 inchSynthesizeraudio-production-session-reimplementationvideoaudio1 video, 2 h 34 min, with sound
Reference answer: 9 synth patchesper-component · mean of components
float tolerance 0.025, seq tolerance 0.02
No task matches that filter.