Where everything ended up
day.mp4 in this directory is a silent first-person recording (1280×720, 10 fps, about 169 minutes, one
continuous take) of one person's day at home. The camera is the resident's eyes. The house has a kitchen, a living
room, a bathroom and two bedrooms. During the day the resident walks between rooms, picks things up and puts them
down in or on furniture and appliances, opens and closes drawers, the fridge, the microwave and the laundry hamper,
fills containers with water at the bathroom sink and pours them out there, cuts food with the knife (eggs get
cracked), opens and closes books, boxes and laptops, and switches phones, laptops and candles on and off.
Nobody else is in the house. Nothing moves or changes state unless the resident does it, and everything the
resident does happens in front of the camera.
Your job: for each of the 60 tracked objects listed below, report where it is and what state it is in at
each of six checkpoints.
Reference material (in reference/)
floorplan.png — top-down plan of the house with room names and furniture labels.
receptacles/<receptacle-id>.jpg — one picture per receptacle, the receptacle tinted magenta, taken at the start
of the recording.
objects/<object-id>.jpg — one picture per tracked object, the object tinted and boxed in magenta, where it is at
the start of the recording.
Some tracked objects are exact look-alikes (the same model): mug-1 / mug-2 / mug-3 / mug-4; cup-1 / cup-2 / cup-3; bowl-1 / bowl-2 / bowl-3; pot-1 / pot-2; apple-1 / apple-2 / apple-3; egg-1 / egg-2 / egg-3; tomato-1 / tomato-2; plate-1 / plate-2 / plate-3; knife-1 / knife-2; book-1 / book-2 / book-3 / book-4; laptop-1 / laptop-2; cellphone-1 / cellphone-2 / cellphone-3; candle-1 / candle-2 / candle-3; box-1 / box-2; remote-1 / remote-2 / remote-3 / remote-4; keychain-1 / keychain-2 / keychain-3; credit-card-1 / credit-card-2; pillow-1 / pillow-2 / pillow-3; spray-bottle-1 / spray-bottle-2 / spray-bottle-3; soap-bottle-1 / soap-bottle-2; vase-1 / vase-2; statue-1 / statue-2. Each ID in such a set is fixed by
where that object is at the start of the recording (its reference picture and its start location below). Keep
track of which one is which through the day.
Checkpoints
Times are positions in day.mp4 (hh:mm:ss from the start of the file). Nothing is being moved or changed within
a few seconds of any checkpoint.
| checkpoint |
video time |
T1 |
00:27:09 |
T2 |
00:55:39 |
T3 |
01:23:41 |
T4 |
01:53:03 |
T5 |
02:21:38 |
END |
02:49:29 (the last frame) |
What to report
location — the ID of the receptacle (from the list below) the object is resting in or on. At every checkpoint
every tracked object rests in or on exactly one listed receptacle (never held, never on the floor). An object on
top of a piece of furniture takes that furniture's ID (e.g. bedroom2-dresser); an object inside one of its
drawers takes the drawer's ID (e.g. bedroom2-dresser-drawer-3). A food item that has been cut stays where it
was cut; its location is the receptacle its pieces lie on.
- State fields, reported only for the objects that list them:
filled — true if the container holds water, otherwise false.
sliced — true once the food item has been cut (for an egg: cracked open); it never goes back to false.
on — true if the device is switched on (phone or laptop screen lit, candle burning), otherwise false.
open — true if the book, box or laptop lid is open, otherwise false.
Drawer and shelf numbers: standing in front of the furniture and facing it, columns are numbered from left to
right, and within a column the numbers run from top to bottom. Each drawer and shelf has its own receptacle picture.
Tracked objects
| object |
state fields |
start location |
mug-1 |
filled |
bathroom-dresser |
mug-2 |
filled |
bedroom2-dresser |
mug-3 |
filled |
bedroom1-dresser |
mug-4 |
filled |
bedroom1-side-table |
cup-1 |
filled |
bedroom2-side-table |
cup-2 |
filled |
kitchen-counter |
cup-3 |
filled |
kitchen-shelf-middle |
bowl-1 |
filled |
bedroom2-table |
bowl-2 |
filled |
bedroom1-desk |
bowl-3 |
filled |
kitchen-table |
pot-1 |
filled |
bedroom2-table |
pot-2 |
filled |
living-table |
apple-1 |
sliced |
living-tv-stand-shelf-3 |
apple-2 |
sliced |
living-tv-stand-shelf-5 |
apple-3 |
sliced |
living-tv-stand |
egg-1 |
sliced |
kitchen-counter |
egg-2 |
sliced |
living-tv-stand-shelf-1 |
egg-3 |
sliced |
kitchen-table |
tomato-1 |
sliced |
kitchen-shelf-top |
tomato-2 |
sliced |
kitchen-shelf-middle |
plate-1 |
— |
kitchen-table |
plate-2 |
— |
living-table |
plate-3 |
— |
kitchen-counter |
knife-1 |
— |
bathroom-dresser |
knife-2 |
— |
kitchen-shelf-top |
book-1 |
open |
bedroom1-bed |
book-2 |
open |
bedroom1-desk |
book-3 |
open |
bedroom2-armchair-2 |
book-4 |
open |
bedroom1-side-table |
laptop-1 |
open, on |
bedroom1-table |
laptop-2 |
open, on |
living-table |
cellphone-1 |
on |
living-sofa |
cellphone-2 |
on |
living-tv-stand-shelf-2 |
cellphone-3 |
on |
bedroom1-bed |
candle-1 |
on |
bedroom1-table |
candle-2 |
on |
living-tv-stand |
candle-3 |
on |
bedroom2-dresser |
box-1 |
open |
bedroom1-table |
box-2 |
open |
bedroom2-table |
remote-1 |
— |
bedroom1-dresser |
remote-2 |
— |
bedroom2-armchair-2 |
remote-3 |
— |
bedroom2-armchair-1 |
remote-4 |
— |
living-tv-stand-shelf-1 |
keychain-1 |
— |
living-tv-stand-shelf-6 |
keychain-2 |
— |
bedroom2-armchair-1 |
keychain-3 |
— |
living-sofa |
credit-card-1 |
— |
living-tv-stand-shelf-2 |
credit-card-2 |
— |
living-tv-stand-shelf-3 |
pillow-1 |
— |
bedroom1-desk |
pillow-2 |
— |
bedroom2-bed |
pillow-3 |
— |
kitchen-counter |
spray-bottle-1 |
— |
bedroom2-side-table |
spray-bottle-2 |
— |
living-tv-stand-shelf-6 |
spray-bottle-3 |
— |
kitchen-table |
soap-bottle-1 |
— |
living-table |
soap-bottle-2 |
— |
living-tv-stand-shelf-4 |
vase-1 |
— |
bedroom1-table |
vase-2 |
— |
bedroom2-table |
statue-1 |
— |
living-tv-stand-shelf-4 |
statue-2 |
— |
bedroom1-desk |
Receptacles (54)
- kitchen:
kitchen-counter, kitchen-fridge, kitchen-microwave, kitchen-table, kitchen-shelf-top, kitchen-shelf-middle, kitchen-shelf-drawer-1, kitchen-shelf-drawer-2
- living:
living-sofa, living-table, living-tv-stand, living-tv-stand-shelf-1, living-tv-stand-shelf-2, living-tv-stand-shelf-3, living-tv-stand-shelf-4, living-tv-stand-shelf-5, living-tv-stand-shelf-6
- bathroom:
bathroom-sink, bathroom-dresser, bathroom-dresser-drawer-1, bathroom-dresser-drawer-2, bathroom-dresser-drawer-3, bathroom-dresser-drawer-4, bathroom-hamper
- bedroom1:
bedroom1-bed, bedroom1-desk, bedroom1-desk-drawer-1, bedroom1-desk-drawer-2, bedroom1-desk-drawer-3, bedroom1-desk-drawer-4, bedroom1-desk-drawer-5, bedroom1-desk-drawer-6, bedroom1-table, bedroom1-dresser, bedroom1-dresser-drawer-1, bedroom1-dresser-drawer-2, bedroom1-dresser-drawer-3, bedroom1-side-table
- bedroom2:
bedroom2-bed, bedroom2-dresser, bedroom2-dresser-drawer-1, bedroom2-dresser-drawer-2, bedroom2-dresser-drawer-3, bedroom2-dresser-drawer-4, bedroom2-dresser-drawer-5, bedroom2-dresser-drawer-6, bedroom2-dresser-drawer-7, bedroom2-dresser-drawer-8, bedroom2-side-table, bedroom2-side-table-drawer-1, bedroom2-side-table-drawer-2, bedroom2-table, bedroom2-armchair-1, bedroom2-armchair-2
Output
Write output/whereabouts.json:
{
"checkpoints": {
"T1": {
"mug-1": {"location": "kitchen-counter", "filled": false},
"apple-1": {"location": "kitchen-fridge", "sliced": false},
"laptop-1": {"location": "living-sofa", "open": false, "on": false},
"knife": {"location": "kitchen-table"}
},
"T2": {},
"T3": {},
"T4": {},
"T5": {},
"END": {}
}
}
(The values above only illustrate the format.) Each checkpoint maps every tracked object ID to an object with
location (a receptacle ID string) and that object's state fields as JSON booleans.
There is a time limit, and whatever is in output/whereabouts.json when the session ends is what counts. Write the
file early and keep updating it as you work.
Rules
- Work only inside this directory. There is no internet access. Do not rely on memory or outside knowledge of
any source material; everything must come from the video and the reference pictures.
- Available tools:
ffmpeg/ffprobe, and Python 3 with numpy, scipy, pandas, pillow,
opencv-python-headless and matplotlib. Nothing else can be installed.