· 2 min read · robotics · computer vision · ros2
Vision-based gesture control for a Kinova Gen3
A webcam, YOLOv8-Pose, a small state machine, and Cartesian velocity commands over ROS2. Wave, and a 7-DOF arm follows.
The system tracks my upper‑body keypoints in real time and turns wrist position into Cartesian velocity commands for the robot.
The problem
Traditional teleoperation needs a bulky controller or a teach pendant. The goal was a hands‑free interface: control a 7‑DOF industrial manipulator with natural body language.
The approach
A vision‑based control loop that bridges computer vision and robotics. A standard RGB webcam detects human pose landmarks (shoulders, wrists, eyes) and maps them to robot actions. Lift a hand or wave, and the Kinova Gen3 moves with low latency.
The pipeline runs on ROS2 in three stages.
Perception (YOLOv8‑Pose). Ultralytics YOLOv8 extracts skeletal keypoints in real time. Unlike a bounding box, pose estimation gives precise (x, y) coordinates for the wrists and shoulders, which is what makes fine‑grained control logic possible.
Decision logic. The system compares keypoints to each other rather than to absolute positions: is the right wrist higher than the right shoulder? Those boolean checks drive a small state machine with states like MOVE_UP, STOP and HOME_POSE.
Actuation (Kortex API). Commands go to the robot through the Kortex API wrapper. To keep motion smooth, the system sends Cartesian velocity vectors instead of absolute position targets, which avoids the jerkiness you get when a continuous gesture is quantised into position steps.
Stack
- Hardware: Kinova Gen3 (7‑DOF), RGB webcam
- Middleware: ROS2
- Vision: Ultralytics YOLOv8‑Pose, OpenCV
- Language and libraries: Python 3.10, Kortex API, NumPy
- Source: github.com/zhuhongd/KINOVA_LAB5
What stayed with me
This started as a standard lab for a robotics course, and it is a student implementation. But it made the idea concrete. Right now I wave a hand and a robot arm in a lab moves. Scale that up and geography stops limiting expertise: a specialist in one city operating in another, an engineer repairing something at the bottom of the sea from a living room. A few hundred lines of code in a lab is how that starts.