Robot Scene 1

A: Bag

B: Paint Bottle

C: Plastic Bowl

D: Container of Metals

E: Table (ignore)

F: Saucer

G: Ceramic Bowl

H: Spoon

I: Pen

J: Pencil

K: Snacks

Here we provide a scene from our real robot evaluation, and all tasks for it. We provide the object detections from OWL-ViT, with color-coded bounding boxes. We provide more descriptive labels for each object detection (these are not what our planner has access to). For each task, we provide videos of the robot executing plans generated using InstructBLIP and PG-InstructBLIP.

Task 1: Move all objects that are not plastic to the side.

Task 2: Move the metal objects into the container with metals.

Task 3: Move all containers that can be used to carry water to the side.

Task 4: Put the two objects with the least mass into the least deformable container.

Task 5: Move the most fragile object to the side.