Codex operating a dual-arm robot
Given natural-language tasks, the model uses tool calls to look, plan, grasp and place: lifting clothes out of a basin, then putting them back. The run includes human hints, recorded below.
- natural-language tasks completed
- 2 / 2
- real time
- 42:36
- strategy changes after failed grasps
- 4
Setup
- Hardware
- Dual robot arms with three live camera feeds.
- Control
- Codex reads state, pulls camera frames and moves joints and grippers through tool calls.
- Task
- Lift clothes out of a basin and set them aside (practiced), then put them back (not specifically practiced).
How the agent works
- 1Perceive
- 2Plan the approach
- 3Close in stages
- 4Test lift
- 5Re-grasp on failure
What we saw
- After a hint, the right arm chose its own viewing angle to support the grasp.
- With the clothes lying flat, it closed the gripper in stages and test-lifted 2 cm before carrying them out.
- When a grasp slipped, the model analyzed why and retried with a new angle and opening.
- Putting the clothes back was direct generalization without specific practice: not neatly placed, but the goal was met.