AgenticVBench-Omni

AgenticVBench-Omni / Meetings and procedures

Cooking

captaincook-recipe-step-transcripts

You are given twenty-two videos in /workspace/materials/. Each is a single continuous recording from a camera worn on the head of one person preparing one dish from start to finish, in a real kitchen. The clips have no audio track. They were recorded by different people in different kitchens, and several of them are the same dish prepared by different people, so recognising the dish does not tell you what happened in a particular clip. Part of the work is deciding, from what you see, which dish each recording is and which of its steps this person actually carried out. Time t is seconds from the start of whichever clip an entry refers to.

Family
Meetings and procedures
Inputs
video
Media
22 videos, 4 h 53 min, no sound track
Reference answer
314 steps
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recallpartial creditper-component

Task package · verifier code

Instruction

Full instruction given to the agent

Egocentric Recipe-Step Transcripts Across 22 Recordings

You are given twenty-two videos in /workspace/materials/. Each is a single continuous recording from a camera worn on the head of one person preparing one dish from start to finish, in a real kitchen. The clips have no audio track. They were recorded by different people in different kitchens, and several of them are the same dish prepared by different people, so recognising the dish does not tell you what happened in a particular clip. Part of the work is deciding, from what you see, which dish each recording is and which of its steps this person actually carried out. Time t is seconds from the start of whichever clip an entry refers to.

video path length time range
A /workspace/materials/A.mp4 15.8 min t = 0 to t = 946.5
B /workspace/materials/B.mp4 11.6 min t = 0 to t = 694.4
C /workspace/materials/C.mp4 13.0 min t = 0 to t = 779.9
D /workspace/materials/D.mp4 13.1 min t = 0 to t = 783.2
E /workspace/materials/E.mp4 17.4 min t = 0 to t = 1044.3
F /workspace/materials/F.mp4 10.9 min t = 0 to t = 652.1
G /workspace/materials/G.mp4 13.5 min t = 0 to t = 809.7
H /workspace/materials/H.mp4 10.1 min t = 0 to t = 608.4
I /workspace/materials/I.mp4 19.2 min t = 0 to t = 1149.6
J /workspace/materials/J.mp4 14.0 min t = 0 to t = 837.4
K /workspace/materials/K.mp4 12.8 min t = 0 to t = 770.4
L /workspace/materials/L.mp4 11.3 min t = 0 to t = 676.5
M /workspace/materials/M.mp4 17.0 min t = 0 to t = 1020.7
N /workspace/materials/N.mp4 10.0 min t = 0 to t = 603.0
O /workspace/materials/O.mp4 11.5 min t = 0 to t = 687.5
P /workspace/materials/P.mp4 15.5 min t = 0 to t = 928.9
Q /workspace/materials/Q.mp4 12.5 min t = 0 to t = 748.8
R /workspace/materials/R.mp4 11.8 min t = 0 to t = 709.2
S /workspace/materials/S.mp4 10.8 min t = 0 to t = 649.8
T /workspace/materials/T.mp4 13.6 min t = 0 to t = 816.2
U /workspace/materials/U.mp4 15.2 min t = 0 to t = 914.6
V /workspace/materials/V.mp4 12.8 min t = 0 to t = 766.0

For each of the twenty-two videos, reconstruct the complete chronological transcript of the recipe steps that person actually performed in that recording, and when each one started and ended.

Nobody here follows the recipe cleanly. Steps are performed out of order, some are performed incorrectly, and some are skipped entirely. Report each recording as it happened, not as the recipe says it should go. A step the person never performed is not an entry, however clearly the recipe calls for it. A step performed twice is two entries. A step performed out of the usual order belongs where it actually happened.

Use any tools in the image, for example ffmpeg and ffprobe, to seek through and sample the videos.

What to submit

Write /workspace/output/solution.json in exactly this shape:

{
  "sequence": [
    {"video": "A", "id": 2, "t_start": 10.0, "t_end": 22.0, "error": "none"},
    {"video": "A", "id": 11, "t_start": 35.0, "t_end": 47.0, "error": "Measurement Error"},
    {"video": "V", "id": 19, "t_start": 60.0, "t_end": 72.0, "error": "Order Error"}
  ]
}

One entry per step performed. Within each video, order the entries by the moment the step begins; if two steps begin at the same second, put the smaller id first. The videos themselves may come in any order. Fields:

  • video: one of "A" through "V", the clip this entry belongs to.
  • id: the step's label, an integer from the closed vocabulary below.
  • t_start: the second at which that step begins, in that clip.
  • t_end: the second at which that same step ends, in that clip.
  • error: how that step was performed, one of the values listed under "How the step was performed" below. Use "none" when the step was carried out as the recipe intends.

An entry counts only if it names the right video, reports how the step was performed, and both of its boundaries land inside the tolerance of the true step. That tolerance is a quarter of the step's duration, never tighter than 1 second and never looser than 3 seconds, and the same tolerance applies to the start and to the end. Short steps are therefore graded strictly at both ends, so watch each action begin and watch it stop.

How the step was performed

These recordings are of people following a recipe, and they do not always follow it correctly. Every entry has to say how its step was performed: "none" if it was done as intended, otherwise the one label below that best describes what went wrong. A step can go wrong in more than one way; name one.

The wording after each label is an invented illustration of that kind of mistake, set in a dish these recordings do not contain. It is not taken from any annotation of them.

  • "none" the step was carried out as the recipe intends.
  • "Measurement Error" for example: two ladles of stock go into the pan where the recipe asks for one.
  • "Missing Step" for example: the barley is never rinsed and goes straight into the simmering pan.
  • "Order Error" for example: the stock is poured in before the lentils rather than after them.
  • "Other" for example: the pan boils over and its rim is wiped down partway through.
  • "Preparation Error" for example: a shallow frying pan stands in for the deep stockpot the recipe names.
  • "Technique Error" for example: the roux is beaten flat with a fork rather than folded with a spatula.
  • "Temperature Error" for example: the burner is left on its lowest setting when a high flame is called for.
  • "Timing Error" for example: the barley is left to simmer four minutes instead of the stated ten.

Deciding this is not the same as recognising the step. The recipe tells you what was supposed to happen; only the video tells you what this person actually did.

The closed vocabulary

One vocabulary covers all twenty-two videos, and it spans 6 different dishes, so most of it belongs to dishes you will not see in any given clip. Use only these labels. Several labels name similar actions on different ingredients, so pick the one that names what the person actually handles.

  • 1 Add-1 tablespoon of olive oil to the mug
  • 2 Add-1 teaspoon of pepper powder to the bowl
  • 3 Add-Add 1 tbsp salsa to the bowl
  • 4 Add-Add 1 teaspoon of softened butter
  • 5 Add-Add 1 teaspoon salt to the bowl
  • 6 Add-Add 1/2 tbsp sweet and sour sauce to the bowl
  • 7 Add-Add basil to the bowl
  • 8 Add-Add chopped cilantro to the bowl
  • 9 Add-Add in 3 tablespoons of milk to the mug
  • 10 Add-Add the corn into a microwave-safe bowl
  • 11 Add-Add the noodles to the bowl
  • 12 Boil-Boil the water. (While the water is boiling, assemble the filter cone)
  • 13 Chop-Chop 1 garlic clove on a cutting board
  • 14 Clean-Clean the knife by wiping it with a paper towel
  • 15 Clean-Clean the knife by wiping with a paper towel
  • 16 Cover-Cover with a lid (or paper towel) to prevent splattering
  • 17 Cross-Cross the floss's two ends over the tortilla roll's top
  • 18 Discard-Discard ends of the tortilla
  • 19 Discard-Discard the paper filter and coffee grounds
  • 20 Extract-Extract lime juice from 1/3 lime
  • 21 Grind-Grind the coffee beans until the coffee grounds are the consistency of coarse sand, about 20 seconds
  • 22 Let-Let the noodles sit for about 1 minute after the microwave stops
  • 23 Measure-Measure 12 ounces of cold water
  • 24 Measure-Measure 2 cups of frozen corn
  • 25 Microwave-Microwave for 1 minute 20 seconds, or until it rises and the toppings are bubbling
  • 26 Microwave-Microwave for 3 minutes, stirring in between
  • 27 Microwave-Microwave the corn for 2 minutes
  • 28 Microwave-Microwave the corn for 3 more minutes
  • 29 Microwave-Microwave the ramen for 4 minutes
  • 30 Mix-Mix in the flavour packet to the bowl
  • 31 Mix-Mix the contents of the bowl well
  • 32 Mix-Mix the contents of the mug thoroughly. (There might be some lumps, but that is ok.)
  • 33 Peel-Peel 1 garlic clove
  • 34 Peel-Peel 1 medium onion
  • 35 Place-Place 8 inch tortilla on a cutting board
  • 36 Place-Place 8-inch flour tortilla on cutting board
  • 37 Place-Place the dripper on top of a coffee mug
  • 38 Place-Place the floss halfway between toothpicks
  • 39 Place-Place the paper filter in the dripper
  • 40 Place-Place the pinwheels on a plate
  • 41 Pour-Pour a small amount of water into the filter to wet the grounds
  • 42 Pour-Pour egg mixture on top of the tortilla
  • 43 Prepare-Prepare the filter insert by folding the paper filter in half to create a semi-circle, and in half again to create a quarter-circle
  • 44 Put-Put all the Vegetables in a microwave-safe bowl
  • 45 Remove-Remove the noodles from the package(Break Noodles / Keep them as a block)
  • 46 Roll-Roll the tortilla from one end to another into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps but not so tight that the filling leaks
  • 47 Roll-Roll the tortilla from one end to the other into a log shape, about 1.5 inches thick. Roll it tight enough to prevent gaps, but not so tight that the filling leaks
  • 48 Secure-Secure the rolled tortilla by inserting 5 toothpicks about 1 inch apart
  • 49 Slide-Slide floss under the tortilla, perpendicular to the length of the roll
  • 50 Spread-Spread jelly over the nut butter
  • 51 Spread-Spread nut butter onto the tortilla, leaving 1/2-inch uncovered at the edges
  • 52 Sprinkle-Sprinkle 1 generous tablespoon of mozzarella cheese on top of the sauce
  • 53 Sprinkle-Sprinkle 1 tbsp shredded cheddar cheese on top of the egg
  • 54 Sprinkle-Sprinkle dried Italian herbs inside the mug
  • 55 Sprinkle-Sprinkle oregano in the bowl
  • 56 Stir-Stir noodles with a spoon or fork until the flavouring dissolves
  • 57 Stir-Stir the contents in the mug well
  • 58 Take-Take 1 tablespoon of marinara sauce
  • 59 Take-Take a microwavable mug
  • 60 Thaw-Thaw the frozen corn by putting it in a sieve and running it under cold water
  • 61 Transfer-Transfer the grounds to the filter cone
  • 62 Trim-Trim the ends of the tortilla roll with the butter knife, leaving 1/2 inch margin between the last toothpick and the end of the roll
  • 63 Wait-Wait about 30 seconds for the coffee to bloom. (You will see small bubbles or foam on the coffee grounds during this step.)
  • 64 Weigh-Weigh the coffee beans (0.8oz-0.12 oz)
  • 65 Whisk-Whisk the egg
  • 66 add-Extract and add contents of an egg to a microwave-safe bowl
  • 67 add-Measure 1/16 teaspoon of baking soda and add it to the mug
  • 68 add-Measure 1/8 teaspoon of baking powder and add it to the mug
  • 69 add-Measure 1/8 teaspoon of salt and add it to the mug
  • 70 add-Measure 4 tablespoons of flour and add it to the mug
  • 71 add-add lime juice to the bowl
  • 72 check-Once the water has boiled, check the temperature of the water. (The water should be between 195-205 degrees Fahrenheit or between 91-96 degrees Celsius. If the water is too hot, let it cool briefly.)
  • 73 cover-cover the noodles with water
  • 74 drain-Let the coffee drain completely into the mug before removing the dripper
  • 75 pour-Slowly pour the rest of the water over the grounds in a circular motion. Do not overfill beyond the top of the paper filter
  • 76 pull-pull the floss ends in opposite directions to slice
  • 77 scoop-Use a butter knife to scoop nut butter from the jar
  • 78 scoop-Use the knife to scoop jelly from the jar
  • 79 slice-slice 1/4 medium onion into pieces
  • 80 slicing-Continue slicing with floss to create 1 more pinwheel
  • 81 spread-spread marinara sauce around the surface of the batter
  • 82 spread-spread open filter in dripper to create a cone
  • 83 stir-then stir the bowl
  • 84 transfer-transfer water to a kettle

Rules

  • Rewrite the complete /workspace/output/solution.json after every ten or so samples you take, not only at the end. Your run can end at any moment and only what is on disk counts, so an incomplete answer saved early beats a complete one you never got to write.
  • Stay inside this working directory. Do not read, write, or search outside it.
  • Do not look anything up online, and do not rely on any memory of these recordings or of the dataset they may come from. Every answer must come from watching these videos.
  • Use only labels from the vocabulary above, and only the error values listed above.
  • An entry with no error field, or with a value not on that list, cannot count.
  • Report every step you find, in every video, including a step performed more than once.

Media

Media files are distributed separately from the repository. Licences named for this task: mit.

Score

AgentScore (0 to 1)
GPT-6-Astra0.105
Claude Opus 5.50.118
Claude Opus 50.035
Gemini 3.8 Flash0.029
DeepSeek 4.1 Flash0.035

One run per agent.

The agent watches 22 continuous head-mounted recordings of people cooking, 293 minutes in all, covering 6 recipes, and writes the transcript of each: every recipe step as it was actually performed, in order, with its start and end and how it was performed (correctly, or with which kind of error). The reference holds 314 steps, 199 of them performed with an error.

What a good answer has to get right, and how the verifier checks it

Each step is named
The step must be the right entry of a fixed list of step descriptions, in the right recording.
Each step is bounded
Both the start and the end must fall within that step’s tolerance: a quarter of the step’s duration, never less than 1 s and never more than 3 s. Short steps are judged strictly and long ones more loosely. The instruction states the same rule.
The order is kept
Within each recording, entries are aligned with the reference in order, so steps listed out of sequence do not all count.
How the step was performed is noticed
The entry must name an error type that the dataset annotates for that step (one is enough where there are several), or “none” for a step performed correctly.
Reciting the recipe does not work
The recordings are not clean executions, and they depart from the usual order of their recipe 77 times. The verifier’s notes report a score of 0.003 for an answer that recites each recipe in its usual order.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 314 steps and precision over every entry in the file, including entries that cannot be used. A step counts only when name, both boundaries and the error field are right.

What scores zero

No solution file or one that cannot be read, an empty list, or a list in which no step is right on every field.

Outside the score: actions that are not steps of the recipe. Entries beyond the first 20,000 are not read. The verifier also reports matches without the error field, on the start alone and on the name alone, as diagnostics.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.12
Wall time
23 min
Steps shown
396

Loading the run…