Open resources

How to compare robot demos with inspectable evidence

To compare two robot demonstrations, give both systems the same task brief and acceptance rule. Count every scheduled attempt, keep assisted completions separate, and link failures to their original records.

RobotBay ·

Define the boundary before testing

For a pick-and-place trial, specify the object, starting pose, lighting, destination and time limit. Define a successful placement and the maximum permitted assistance. Record hardware and software versions. Run changed conditions as a separate series.

Illustrative comparison · 10 attempts per system

Suppose A finishes eight attempts autonomously and two with help. B finishes seven autonomously, two with help and one unsuccessfully. Overall completions are 10/10 and 9/10. Autonomous completions are 8/10 and 7/10. Keep the unsuccessful attempt in the denominator. These invented numbers illustrate the calculation only.

Record failure and recovery

For a dropped object, log the attempt ID, timestamp, last completed step, reset or operator action, recovery duration and evidence link. Write the boundary as a testable sentence: this result covers the tested object and lighting; repeat the trial before applying it to a different surface.