AgenticVBench-Omni

AgenticVBench-Omni / Meetings and procedures

Robot Recovery

robot-fault-recovery

You are taking over a robot arm mid-task from a previous operator who was interrupted partway through a checklist. You control the arm through the API in robot.py, in this directory. You have no history: no log of what the previous operator did, no note about what went wrong. The only way to find out what has and hasn't been done -- and whether anything went wrong -- is to look at the scene yourself and reason about it.

Family
Meetings and procedures
Inputs
interactive simulator
Media
a live robot simulator with two cameras; no media files
Reference answer
8 checklist steps
Deliverable
/workspace/output
Scoring
per-componentchecklist / accuracygated productweighted

Task package · verifier code

Instruction

Full instruction given to the agent

Interrupted tidy-up: diagnose and finish

You are taking over a robot arm mid-task from a previous operator who was interrupted partway through a checklist. You control the arm through the API in robot.py, in this directory. You have no history: no log of what the previous operator did, no note about what went wrong. The only way to find out what has and hasn't been done -- and whether anything went wrong -- is to look at the scene yourself and reason about it.

The scene

A drawer chest sits to your right. To your left, along the counter, sit: a green tray, an orange tray, a small block, a can, a mug, a plate, and two bowls. The exterior camera shows the whole counter and the front of the chest. The wrist camera can be moved in close to look inside the drawer or the trays.

Checklist

The previous operator was working through this list, in order, before being pulled away. Some number of steps (you don't know how many) were genuinely finished; at most one thing that looks finished, or partly finished, is not actually in the state it should be. Nothing in this prompt says which steps were done or what -- if anything -- is wrong. Work it out from the cameras.

  1. Open the drawer of the chest.
  2. Put the block into the drawer.
  3. Close the drawer.
  4. Move the mug to a clear spot on the counter, away from everything else.
  5. Stack the second bowl on top of the first bowl.
  6. Move the plate to a clear spot on the counter, away from everything else.
  7. Put the can into the green tray. (The orange tray is never a destination for anything on this checklist -- if something ends up there, that's not what was asked.)
  8. Finish: park the gripper at the rest point, x = -0.05, y = 0.00, z = 1.30 (within 0.10 m of it horizontally and vertically), clear of the chest, and open the gripper. home() does not reach the rest point: it stops at about z = 1.01, well below it, so finish with a move_ee.

A step only counts as done if it is still true when you finish -- if something you find already correct gets disturbed later (by you or by anything else), that step is no longer done. Likewise, if something is genuinely broken, it isn't fixed just by looking at it: fix it and leave it fixed.

API

robot.py is a small CLI and importable client. Every call prints one JSON object and also refreshes two image files you should look at before deciding your next move: output/obs/exterior.png (a fixed external view of the whole counter and the chest) and output/obs/wrist.png (a camera mounted on the gripper, for close-up and occluded views -- move the gripper near something and look down to see inside a container or a gap).

python3 robot.py observe
python3 robot.py move_ee --x 0.15 --y -0.10 --z 0.95 --yaw 0.0
python3 robot.py move_delta --dx 0 --dy 0 --dz -0.05 --dyaw 0
python3 robot.py gripper --action close       # or --action open
python3 robot.py home                         # joint-space reset; gripper must be open first
  • move_ee(x, y, z, yaw): move the gripper to an absolute position (meters) and yaw (radians); the gripper always points straight down, only yaw is free.
  • move_delta(dx, dy, dz, dyaw): move relative to the current position.
  • gripper(open|close): open or close the gripper.
  • home(): snap the arm to its one canonical joint configuration (not a move to a position you choose). move_ee/move_delta only constrain the gripper's own pose, so a redundant joint can drift into an awkward configuration that makes a later, otherwise-reachable target fail repeatedly for no visible reason -- if a pick keeps missing something that looks perfectly reachable, try home() before the next attempt. Requires the gripper already open. Not the checklist's rest point (step 8).
  • observe(): look, without spending a step.

Every response includes observation.proprio (ee_pos, ee_yaw, gripper_width, wrist_force, wrist_torque) and observation.steps_remaining. observe() is free; the other four each cost one step. You have a budget of 150 steps for the whole episode. A response with "ok": false means that call did not count as progress (e.g. a malformed request) -- read error and adjust; once the budget is spent, every further action call returns budget_exhausted and does nothing. Moving something you have already picked up costs the same one step as any other move -- plan pick-and-place sequences (a lift, a lateral move, a descent) rather than a single diagonal move through the workspace, which can knock over something else on the way.

Output

Before you finish, write your own record of what you believe is true at the end to output/done.json, one boolean per checklist step, e.g.:

{"1_drawer_open": true, "2_block_in_drawer": true, "3_drawer_closed": true,
 "4_mug_clear": true, "5_bowls_stacked": true, "6_plate_clear": true,
 "7_can_in_green_tray": true, "8_finished": true}

This file is a record of your own assessment; it is not what is scored (the actual outcome is read from the robot state, not from your report).

Rules

  • Work only inside this directory. No internet access is available.
  • The only way to see or affect the scene is through robot.py. Do not try to inspect the simulator process, its files, or any other process on the machine -- there is nothing there for you, and doing so is treated as leaving the task.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 1 files (environment/media/SHA256SUMS.txt). Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.967
Claude Opus 5.50.967
Claude Opus 50.830
Gemini 3.8 Flash0.150
DeepSeek 4.1 Flash0.150

One run per agent.

The agent takes over a simulated robot arm partway through an eight-step tidy-up checklist, with no record of what the previous operator did. Through two cameras it has to work out which steps are finished and which one was left in a wrong state, then repair that and complete the list, within a budget of 150 actions. What is scored is the state of the scene in the simulator’s log, not the agent’s written account.

What a good answer has to get right, and how the verifier checks it

The fault is found and repaired
The step the previous operator left in a wrong state has to end in its correct state. This part is worth 0.8 of the score. Carrying out the checklist step in the ordinary way does not clear the fault.
The rest of the list is finished
Steps the previous operator never reached, and the final return of the arm to its rest position, are worth 0.1 together.
A step counts only if it stays done
Each step is checked for being reached at some point and for still holding at the end, with further checks on position and orientation. A placement counts only at a moment when the gripper is not holding the object, so carrying something through its target does not count.
Steps that were already done earn nothing
A step that is already complete before the agent’s first action gives no credit, so doing nothing scores 0.
Nothing that was fine is broken
The last 0.1 is for leaving the already-finished steps intact: it is reduced by the share of them that are no longer true at the end, and scaled by how much of the real work was done.

How the checks become one score

Score = 0.8 × recovery + 0.1 × ordinary + 0.1 × no-regression × engagement, where engagement is the mean of the recovery and ordinary parts. A run that finishes the ordinary steps and never deals with the fault scores at most 0.15.

What scores zero

A missing or malformed simulator log, or a run that neither repairs the fault nor completes any remaining step.

Outside the score: the assessment file the agent writes about what it found, which the instruction says is a record and is not scored, and the order in which the steps are done.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
REST_TOL0.1constant in the verifier code
POS_TOL0.05constant in the verifier code
VERT_TOL0.08constant in the verifier code
DRAWER_JAM_MARGIN0.015constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.97
Wall time
15.8 min
Steps shown
79

Loading the run…