AgenticVBench-Omni

AgenticVBench-Omni / Spatial measurement and tracking

House Walkthrough

house-walkthrough-floorplan-inventory

walkthrough.mp4 in your working directory is one continuous, roughly 55-minute first-person walkthrough of a single-storey house, filmed by a hand-held camera carried through it at walking pace. The house is computer-rendered, but treat it as a real building with real metric dimensions. The walk enters every room several times, but a single visit usually shows only part of a room. Closed doors stay closed and are never passed through. The video has no audio.

Family
Spatial measurement and tracking
Inputs
videodocuments
Media
1 video, 54 min, no sound track
Reference answer
10 rooms, 10 doorways and 125 objects
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matchinggated productgeometric (IoU / distance)

Task package · verifier code

Instruction

Full instruction given to the agent

House Floor Plan and Inventory from a Walkthrough

walkthrough.mp4 in your working directory is one continuous, roughly 55-minute first-person walkthrough of a single-storey house, filmed by a hand-held camera carried through it at walking pace. The house is computer-rendered, but treat it as a real building with real metric dimensions. The walk enters every room several times, but a single visit usually shows only part of a room. Closed doors stay closed and are never passed through. The video has no audio.

From this video, reconstruct the floor plan of the whole house (every room's type and outline, and every doorway with its position) and the inventory of the objects listed below, with each object's room and position.

Camera

  • 1280×720 pixels, 15 frames per second, ideal pinhole camera with no lens distortion.
  • Focal length 937.83 px horizontally and vertically (vertical field of view 42°); principal point at pixel (640, 360), the image centre. These never change.
  • The optical centre is always 1.576 m above the floor. The floor is flat and level throughout the house.
  • The camera never rolls: the image's horizontal axis always stays parallel to the floor. Its heading and its downward tilt change smoothly as it is carried around; the tilt stays between 0° (looking horizontally) and 30° below horizontal. Neither is given per frame.

Coordinate frame

Report every coordinate in metres on the floor plane, viewed from above:

  • Origin: the floor point directly below the camera in the first frame.
  • +y: the camera's viewing direction in the first frame, projected onto the floor.
  • +x: horizontal, 90° clockwise from +y seen from above, i.e. toward the right-hand side of the first frame.

What to report

Rooms. Every room of the house, each exactly once. A room is a floor area enclosed by walls; doorways join rooms but do not split or merge them. Each room is exactly one of: kitchen, living_room, bedroom, bathroom. Its outline is the polygon traced by the room's walls along the floor (the line where floor meets wall), continuing straight across doorways; vertices in order around the room (either direction), without repeating the first vertex. The number of rooms you report must equal the true number of rooms in the house; a plan with the wrong room count is not accepted. A room counts as correct when its type is right and the area overlap between your outline and the true outline is at least 0.85 of their combined area (intersection over union ≥ 0.85).

Doors. Every doorway in a wall through which a person could walk — with or without a door leaf, open or closed — including doors in outer walls. Report each doorway once, with the pair of rooms on its two sides (for a door in an outer wall, the second side is "outside") and its position: the centre of the opening, i.e. the midpoint of the gap in the wall, on the floor–wall line. Windows are not doors. A door counts as correct when it joins the right pair of rooms and its position is within 0.25 m of the true opening centre.

Objects. Every instance, anywhere in the house, of the classes below — furniture, fixtures, and the listed smaller items wherever they are (on the floor, on furniture, or on a wall). Use exactly these labels and report nothing else. Not every class occurs in this house.

label what it is
bed any bed, including bunk and loft beds
sofa upholstered seat for two or more people
armchair upholstered chair for one person, with arms
chair_or_stool any other single seat: dining chair, desk chair, stool
table_or_cabinet any table (dining table, desk, side table, nightstand, coffee table), kitchen counter unit, or free-standing storage furniture (dresser, chest of drawers, cabinet, TV stand, shelving unit)
television flat-screen TV, on furniture or mounted on a wall
floor_lamp tall lamp standing on the floor
desk_lamp small lamp standing on furniture
fridge refrigerator
microwave microwave oven
toaster toaster
coffee_machine coffee machine
sink washbasin, including its pedestal or cabinet
toilet toilet
garbage_can waste bin (not a loose trash bag)
laundry_hamper lidded basket for laundry
laundry_machine washing machine or clothes dryer
safe small safe (lockable strongbox)
dog_bed pet bed on the floor
houseplant potted plant, any size
painting framed picture or poster hanging on a wall
window window in a wall
pillow pillow or cushion
teddy_bear stuffed toy bear
basketball basketball
baseball_bat baseball bat
tennis_racket tennis racket
laptop laptop computer
book book
newspaper newspaper
box cardboard box
statue decorative figurine or sculpture
candle candle, with or without a holder
alarm_clock alarm clock
watch wristwatch
cell_phone mobile phone
remote_control remote control
credit_card credit card
cd compact disc
key_chain keys on a ring
pen_or_pencil pen or pencil
bowl_or_vase bowl, dish or vase, open or lidded
plate plate
mug_or_cup mug or cup
bottle bottle (not a spray or soap bottle)
salt_or_pepper_shaker salt or pepper shaker
food loose food item: apple, bread, egg, lettuce, potato, tomato
utensil fork, spoon, knife, ladle or spatula
cookware pan, pot or kettle
plunger toilet plunger
toilet_paper roll of toilet paper
paper_towel_roll roll of paper towels
tissue_box box of tissues
spray_bottle spray bottle
soap_bottle soap dispenser bottle
soap_bar bar of soap
dish_sponge dish sponge

For each object give its room (the room whose outline contains it) and its position: the centre of its footprint, i.e. the centre of the rectangle it covers when seen from directly above (for an object on a wall, the rectangle directly below it on the floor). An object counts as correct when its label and room are right and its position is within 0.25 m of the true footprint centre.

Output

Write output/floorplan.json:

{
  "rooms": [
    {"id": "r1", "type": "kitchen", "outline": [[20.0, 20.0], [20.0, 23.5], [23.5, 23.5], [23.5, 20.0]]},
    {"id": "r2", "type": "bedroom", "outline": [[23.5, 20.0], [23.5, 23.5], [26.5, 23.5], [26.5, 20.0]]}
  ],
  "doors": [
    {"rooms": ["r1", "r2"], "position": [23.5, 21.2]},
    {"rooms": ["r1", "outside"], "position": [21.0, 20.0]}
  ],
  "objects": [
    {"class": "bed", "room": "r2", "position": [25.0, 22.0]},
    {"class": "fridge", "room": "r1", "position": [20.5, 23.0]}
  ]
}

Room ids are any unique strings you choose (not "outside"); doors and objects refer to them. The values above only illustrate the format.

The session has a time limit and whatever is in output/floorplan.json when it ends is what is graded, so write a complete file as soon as you have a first full plan and keep overwriting it as you improve it.

Rules

  • Work only inside your working directory. There is no internet access; do not try to install packages.
  • Use only what you observe in walkthrough.mp4. Do not rely on memory of any dataset, simulator, or house layout.
  • Available tools are what is installed: ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib.

Media

Frame from walkthrough.mp4
walkthrough.mp4 · 817 s
Frame from walkthrough.mp4
walkthrough.mp4 · 1635 s
Frame from walkthrough.mp4
walkthrough.mp4 · 2452 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 1 files (environment/media/SHA256SUMS.txt). Licences named for this task: see NOTICE.

Score

AgentScore (0 to 1)
GPT-6-Astra0.290
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a 55-minute silent walkthrough of a single-storey house and draws it: the outline and type of every room in metres, every doorway, and the position of every object of a listed set of classes. The reference holds 10 rooms, 10 doorways and 125 objects.

What a good answer has to get right, and how the verifier checks it

The house has the right number of rooms
The number of rooms reported must equal the true number, 10. The instruction states that a plan with the wrong room count is not accepted.
Each room has the right shape and type
Rooms are paired with the true rooms so that total overlap is largest. A room is correct when its type matches and its outline overlaps the true outline with intersection over union of at least 0.85. The room score is the share of rooms that are correct.
Doorways join the right rooms
A doorway counts when it joins the same pair of rooms, or a room and the outside, and its centre is within 0.25 m. The result is an F1 over doorways.
Objects are found, classed and placed
An object counts when its class and its room are right and its position is within 0.25 m, one to one. The result is an F1 over the 125 objects.
Objects seen only in passing are not held against the answer
29 objects that the video shows only fleetingly are excused: reporting one is left out of the count and is never rewarded.

How the checks become one score

Score = room score × doorway F1 × object F1. The three parts are multiplied, so the score cannot be higher than the weakest part: a correct plan with half the objects found scores about two thirds, and a full inventory in a wrongly drawn plan scores little, because objects and doorways are checked through the room pairing.

What scores zero

No answer or one that cannot be read, a room count other than 10, no room correct, no doorway matched, or no object matched.

Outside the score: the 29 excused objects, windows, and anything not in the listed object classes.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.29
Wall time
67.7 min
Steps shown
188

Loading the run…