AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

Object Whereabouts

object-whereabouts-day-ledger

day.mp4 in this directory is a silent first-person recording (1280×720, 10 fps, about 169 minutes, one continuous take) of one person's day at home. The camera is the resident's eyes. The house has a kitchen, a living room, a bathroom and two bedrooms. During the day the resident walks between rooms, picks things up and puts them down in or on furniture and appliances, opens and closes drawers, the fridge, the microwave and the laundry hamper, fills containers with water at the bathroom sink and pours them out there, cuts food with the knife (eggs get cracked), opens and closes books, boxes and laptops, and switches phones, laptops and candles on and off.

Family
Spatial measurement and tracking
Inputs
videodocuments
Media
1 video, 2 h 50 min, no sound track
Reference answer
302 scored answers for 57 objects
Deliverable
/workspace/output
Scoring
checklist / accuracy

Task package · verifier code

Instruction

Full instruction given to the agent

Where everything ended up

day.mp4 in this directory is a silent first-person recording (1280×720, 10 fps, about 169 minutes, one continuous take) of one person's day at home. The camera is the resident's eyes. The house has a kitchen, a living room, a bathroom and two bedrooms. During the day the resident walks between rooms, picks things up and puts them down in or on furniture and appliances, opens and closes drawers, the fridge, the microwave and the laundry hamper, fills containers with water at the bathroom sink and pours them out there, cuts food with the knife (eggs get cracked), opens and closes books, boxes and laptops, and switches phones, laptops and candles on and off.

Nobody else is in the house. Nothing moves or changes state unless the resident does it, and everything the resident does happens in front of the camera.

Your job: for each of the 60 tracked objects listed below, report where it is and what state it is in at each of six checkpoints.

Reference material (in reference/)

  • floorplan.png — top-down plan of the house with room names and furniture labels.
  • receptacles/<receptacle-id>.jpg — one picture per receptacle, the receptacle tinted magenta, taken at the start of the recording.
  • objects/<object-id>.jpg — one picture per tracked object, the object tinted and boxed in magenta, where it is at the start of the recording.

Some tracked objects are exact look-alikes (the same model): mug-1 / mug-2 / mug-3 / mug-4; cup-1 / cup-2 / cup-3; bowl-1 / bowl-2 / bowl-3; pot-1 / pot-2; apple-1 / apple-2 / apple-3; egg-1 / egg-2 / egg-3; tomato-1 / tomato-2; plate-1 / plate-2 / plate-3; knife-1 / knife-2; book-1 / book-2 / book-3 / book-4; laptop-1 / laptop-2; cellphone-1 / cellphone-2 / cellphone-3; candle-1 / candle-2 / candle-3; box-1 / box-2; remote-1 / remote-2 / remote-3 / remote-4; keychain-1 / keychain-2 / keychain-3; credit-card-1 / credit-card-2; pillow-1 / pillow-2 / pillow-3; spray-bottle-1 / spray-bottle-2 / spray-bottle-3; soap-bottle-1 / soap-bottle-2; vase-1 / vase-2; statue-1 / statue-2. Each ID in such a set is fixed by where that object is at the start of the recording (its reference picture and its start location below). Keep track of which one is which through the day.

Checkpoints

Times are positions in day.mp4 (hh:mm:ss from the start of the file). Nothing is being moved or changed within a few seconds of any checkpoint.

checkpoint video time
T1 00:27:09
T2 00:55:39
T3 01:23:41
T4 01:53:03
T5 02:21:38
END 02:49:29 (the last frame)

What to report

  • location — the ID of the receptacle (from the list below) the object is resting in or on. At every checkpoint every tracked object rests in or on exactly one listed receptacle (never held, never on the floor). An object on top of a piece of furniture takes that furniture's ID (e.g. bedroom2-dresser); an object inside one of its drawers takes the drawer's ID (e.g. bedroom2-dresser-drawer-3). A food item that has been cut stays where it was cut; its location is the receptacle its pieces lie on.
  • State fields, reported only for the objects that list them:
  • filled — true if the container holds water, otherwise false.
  • sliced — true once the food item has been cut (for an egg: cracked open); it never goes back to false.
  • on — true if the device is switched on (phone or laptop screen lit, candle burning), otherwise false.
  • open — true if the book, box or laptop lid is open, otherwise false.

Drawer and shelf numbers: standing in front of the furniture and facing it, columns are numbered from left to right, and within a column the numbers run from top to bottom. Each drawer and shelf has its own receptacle picture.

Tracked objects

object state fields start location
mug-1 filled bathroom-dresser
mug-2 filled bedroom2-dresser
mug-3 filled bedroom1-dresser
mug-4 filled bedroom1-side-table
cup-1 filled bedroom2-side-table
cup-2 filled kitchen-counter
cup-3 filled kitchen-shelf-middle
bowl-1 filled bedroom2-table
bowl-2 filled bedroom1-desk
bowl-3 filled kitchen-table
pot-1 filled bedroom2-table
pot-2 filled living-table
apple-1 sliced living-tv-stand-shelf-3
apple-2 sliced living-tv-stand-shelf-5
apple-3 sliced living-tv-stand
egg-1 sliced kitchen-counter
egg-2 sliced living-tv-stand-shelf-1
egg-3 sliced kitchen-table
tomato-1 sliced kitchen-shelf-top
tomato-2 sliced kitchen-shelf-middle
plate-1 — kitchen-table
plate-2 — living-table
plate-3 — kitchen-counter
knife-1 — bathroom-dresser
knife-2 — kitchen-shelf-top
book-1 open bedroom1-bed
book-2 open bedroom1-desk
book-3 open bedroom2-armchair-2
book-4 open bedroom1-side-table
laptop-1 open, on bedroom1-table
laptop-2 open, on living-table
cellphone-1 on living-sofa
cellphone-2 on living-tv-stand-shelf-2
cellphone-3 on bedroom1-bed
candle-1 on bedroom1-table
candle-2 on living-tv-stand
candle-3 on bedroom2-dresser
box-1 open bedroom1-table
box-2 open bedroom2-table
remote-1 — bedroom1-dresser
remote-2 — bedroom2-armchair-2
remote-3 — bedroom2-armchair-1
remote-4 — living-tv-stand-shelf-1
keychain-1 — living-tv-stand-shelf-6
keychain-2 — bedroom2-armchair-1
keychain-3 — living-sofa
credit-card-1 — living-tv-stand-shelf-2
credit-card-2 — living-tv-stand-shelf-3
pillow-1 — bedroom1-desk
pillow-2 — bedroom2-bed
pillow-3 — kitchen-counter
spray-bottle-1 — bedroom2-side-table
spray-bottle-2 — living-tv-stand-shelf-6
spray-bottle-3 — kitchen-table
soap-bottle-1 — living-table
soap-bottle-2 — living-tv-stand-shelf-4
vase-1 — bedroom1-table
vase-2 — bedroom2-table
statue-1 — living-tv-stand-shelf-4
statue-2 — bedroom1-desk

Receptacles (54)

  • kitchen: kitchen-counter, kitchen-fridge, kitchen-microwave, kitchen-table, kitchen-shelf-top, kitchen-shelf-middle, kitchen-shelf-drawer-1, kitchen-shelf-drawer-2
  • living: living-sofa, living-table, living-tv-stand, living-tv-stand-shelf-1, living-tv-stand-shelf-2, living-tv-stand-shelf-3, living-tv-stand-shelf-4, living-tv-stand-shelf-5, living-tv-stand-shelf-6
  • bathroom: bathroom-sink, bathroom-dresser, bathroom-dresser-drawer-1, bathroom-dresser-drawer-2, bathroom-dresser-drawer-3, bathroom-dresser-drawer-4, bathroom-hamper
  • bedroom1: bedroom1-bed, bedroom1-desk, bedroom1-desk-drawer-1, bedroom1-desk-drawer-2, bedroom1-desk-drawer-3, bedroom1-desk-drawer-4, bedroom1-desk-drawer-5, bedroom1-desk-drawer-6, bedroom1-table, bedroom1-dresser, bedroom1-dresser-drawer-1, bedroom1-dresser-drawer-2, bedroom1-dresser-drawer-3, bedroom1-side-table
  • bedroom2: bedroom2-bed, bedroom2-dresser, bedroom2-dresser-drawer-1, bedroom2-dresser-drawer-2, bedroom2-dresser-drawer-3, bedroom2-dresser-drawer-4, bedroom2-dresser-drawer-5, bedroom2-dresser-drawer-6, bedroom2-dresser-drawer-7, bedroom2-dresser-drawer-8, bedroom2-side-table, bedroom2-side-table-drawer-1, bedroom2-side-table-drawer-2, bedroom2-table, bedroom2-armchair-1, bedroom2-armchair-2

Output

Write output/whereabouts.json:

{
  "checkpoints": {
    "T1": {
      "mug-1": {"location": "kitchen-counter", "filled": false},
      "apple-1": {"location": "kitchen-fridge", "sliced": false},
      "laptop-1": {"location": "living-sofa", "open": false, "on": false},
      "knife": {"location": "kitchen-table"}
    },
    "T2": {},
    "T3": {},
    "T4": {},
    "T5": {},
    "END": {}
  }
}

(The values above only illustrate the format.) Each checkpoint maps every tracked object ID to an object with location (a receptacle ID string) and that object's state fields as JSON booleans.

There is a time limit, and whatever is in output/whereabouts.json when the session ends is what counts. Write the file early and keep updating it as you work.

Rules

  • Work only inside this directory. There is no internet access. Do not rely on memory or outside knowledge of any source material; everything must come from the video and the reference pictures.
  • Available tools: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless and matplotlib. Nothing else can be installed.

Media

Frame from day.mp4
day.mp4 · 2542 s
Frame from day.mp4
day.mp4 · 5085 s
Frame from day.mp4
day.mp4 · 7627 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 116 files (environment/media/SHA256SUMS.txt). Licences named for this task: see NOTICE.

Score

AgentScore (0 to 1)
GPT-6-Astra0.035
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a silent 169-minute first-person recording of one person’s day at home and, at six checkpoints, states for each of 60 tracked objects which piece of furniture or appliance it is in or on, and its state (for example open, filled, switched on). Several objects are exact look-alikes, such as four identical mugs.

What a good answer has to get right, and how the verifier checks it

Only changes are scored
A checkpoint answer is scored only when the true answer for that object differs from the previous checkpoint, or from the starting state at the first one. Repeating the starting state for everything therefore earns nothing. There are 302 such answers, for 57 objects.
The location is exact
The location must be the exact receptacle ID, such as a particular drawer or shelf; the right room or the right piece of furniture with the wrong shelf is wrong.
The state is right
Every state field of the object must be given as true or false and match.
Each object is followed through the whole day
An object counts only when all of its scored answers are right, at every checkpoint where it changed. One wrong checkpoint loses the object.

How the checks become one score

Score = objects with every scored answer right / 57. Objects weigh the same whether they changed once or at all six checkpoints. Mixing up two look-alike mugs loses both.

What scores zero

No whereabouts.json or one that cannot be read, or an answer in which no object is right at all of its scored checkpoints.

Outside the score: answers for an object at checkpoints where it has not changed since the previous one, and the 3 tracked objects that never change.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.04
Wall time
20.3 min
Steps shown
102

Loading the run…