The agent gets three first-person clips of a person handling small objects on a table. In each clip one object is designated, and for a set of query frames the agent gives its full pose in that clip’s camera coordinates: position in metres and orientation as a quaternion. The reference holds 36 query frames.
What a good answer has to get right, and how the verifier checks it
- Position and orientation are both right
- The verifier moves the object’s reference point set by the reported pose and by the true pose and averages the distance between corresponding points (the ADD measure used in pose-estimation benchmarks). A wrong rotation and a wrong depth both show up as distance.
- Closer is better, up to a tenth of the object’s size
- Each frame scores 1 − average distance / (10% of the object’s diameter): 1.0 for an exact pose, 0 once the average distance reaches a tenth of the diameter.
- Symmetry is allowed for
- For an object flagged as symmetric each point is compared with the nearest true point instead, so a rotation that leaves the object looking the same is not penalized. None of the three objects here is flagged.
- Every query frame is answered
- A missing or malformed frame scores 0 and still counts in the mean.
How the checks become one score
Score = the mean frame score over all query frames of the three clips. A near miss earns part of the credit: there is no single pass-or-fail distance.
What scores zero
No answer or one that cannot be read, or an answer in which every pose is off by a tenth of the object’s diameter or more.
Outside the score: any frame that is not a query frame, and the other objects on the table.