Interrupted tidy-up: diagnose and finish
You are taking over a robot arm mid-task from a previous operator who was interrupted partway
through a checklist. You control the arm through the API in robot.py, in this directory. You
have no history: no log of what the previous operator did, no note about what went wrong. The
only way to find out what has and hasn't been done -- and whether anything went wrong -- is to
look at the scene yourself and reason about it.
The scene
A drawer chest sits to your right. To your left, along the counter, sit: a green tray, an
orange tray, a small block, a can, a mug, a plate, and two bowls. The exterior camera shows the
whole counter and the front of the chest. The wrist camera can be moved in close to look inside
the drawer or the trays.
Checklist
The previous operator was working through this list, in order, before being pulled away. Some
number of steps (you don't know how many) were genuinely finished; at most one thing that looks
finished, or partly finished, is not actually in the state it should be. Nothing in this prompt
says which steps were done or what -- if anything -- is wrong. Work it out from the cameras.
- Open the drawer of the chest.
- Put the block into the drawer.
- Close the drawer.
- Move the mug to a clear spot on the counter, away from everything else.
- Stack the second bowl on top of the first bowl.
- Move the plate to a clear spot on the counter, away from everything else.
- Put the can into the green tray. (The orange tray is never a destination for anything on
this checklist -- if something ends up there, that's not what was asked.)
- Finish: park the gripper at the rest point, x = -0.05, y = 0.00, z = 1.30 (within 0.10 m of
it horizontally and vertically), clear of the chest, and open the gripper.
home() does not
reach the rest point: it stops at about z = 1.01, well below it, so finish with a move_ee.
A step only counts as done if it is still true when you finish -- if something you find already
correct gets disturbed later (by you or by anything else), that step is no longer done. Likewise,
if something is genuinely broken, it isn't fixed just by looking at it: fix it and leave it fixed.
API
robot.py is a small CLI and importable client. Every call prints one JSON object and also
refreshes two image files you should look at before deciding your next move:
output/obs/exterior.png (a fixed external view of the whole counter and the chest) and
output/obs/wrist.png (a camera mounted on the gripper, for close-up and occluded views --
move the gripper near something and look down to see inside a container or a gap).
python3 robot.py observe
python3 robot.py move_ee --x 0.15 --y -0.10 --z 0.95 --yaw 0.0
python3 robot.py move_delta --dx 0 --dy 0 --dz -0.05 --dyaw 0
python3 robot.py gripper --action close # or --action open
python3 robot.py home # joint-space reset; gripper must be open first
move_ee(x, y, z, yaw): move the gripper to an absolute position (meters) and yaw (radians);
the gripper always points straight down, only yaw is free.
move_delta(dx, dy, dz, dyaw): move relative to the current position.
gripper(open|close): open or close the gripper.
home(): snap the arm to its one canonical joint configuration (not a move to a position you
choose). move_ee/move_delta only constrain the gripper's own pose, so a redundant joint
can drift into an awkward configuration that makes a later, otherwise-reachable target fail
repeatedly for no visible reason -- if a pick keeps missing something that looks perfectly
reachable, try home() before the next attempt. Requires the gripper already open. Not
the checklist's rest point (step 8).
observe(): look, without spending a step.
Every response includes observation.proprio (ee_pos, ee_yaw, gripper_width,
wrist_force, wrist_torque) and observation.steps_remaining. observe() is free; the other
four each cost one step. You have a budget of 150 steps for the whole episode. A response
with "ok": false means that call did not count as progress (e.g. a malformed request) --
read error and adjust; once the budget is spent, every further action call returns
budget_exhausted and does nothing. Moving something you have already picked up costs the same
one step as any other move -- plan pick-and-place sequences (a lift, a lateral move, a
descent) rather than a single diagonal move through the workspace, which can knock over
something else on the way.
Output
Before you finish, write your own record of what you believe is true at the end to
output/done.json, one boolean per checklist step, e.g.:
{"1_drawer_open": true, "2_block_in_drawer": true, "3_drawer_closed": true,
"4_mug_clear": true, "5_bowls_stacked": true, "6_plate_clear": true,
"7_can_in_green_tray": true, "8_finished": true}
This file is a record of your own assessment; it is not what is scored (the actual outcome is
read from the robot state, not from your report).
Rules
- Work only inside this directory. No internet access is available.
- The only way to see or affect the scene is through
robot.py. Do not try to inspect the
simulator process, its files, or any other process on the machine -- there is nothing there
for you, and doing so is treated as leaving the task.