SawyerSense is the system I built for my master’s thesis at CSU Channel Islands: an intelligent human-robot interaction system that lets people direct a Sawyer collaborative robot with their voice.
When it starts, the arm moves to a photo position and an Intel RealSense camera captures the workspace. GPT-4o handles object detection, recognizing the shapes and colors on the table, while a GPT service turns spoken commands into grab-and-place actions. The robot answers out loud through text-to-speech.
Sawyer runs on ROS1, which isn’t compatible with modern AI libraries out of the box. I wrote a FastAPI bridge between the two so the language and vision services could drive the robot, and added sensor-based retry logic for when the gripper misses a grab.
The GPT and vision services run as separate FastAPI apps alongside the ROS launch files, and the ROS deployment is containerized with Docker. I also experimented with several LLMs to balance reasoning quality against latency.
Built with
- Python
- ROS
- Intera SDK
- OpenAI GPT-4o
- FastAPI
- Intel RealSense
- Docker