HAND
Current Project — 2026 · In Progress

The Hand

Status
In Progress
Year
2026
Type
Mechanical · Electrical · Software
Stack
ROS2 · YOLOv8 · Whisper · SolidWorks · Altium · ESP32
// 01 — What It Is

Accessible Chess Arm

The Hand is an autonomous 6DOF robotic arm that lets people with motor disabilities play physical chess through voice control and computer vision — fully hands-free.

Many people living with conditions like ALS, muscular dystrophy, or severe arthritis love chess but physically can't manipulate the pieces. Existing solutions push them to digital-only play, removing the social experience of sitting across a real board from another person. The Hand bridges that gap — a voice command triggers the arm to pick up and move the correct piece on a real board, while YOLOv8 tracks the full board state in real time to confirm every move.

This project was inspired by the countless 4DOF robotic arms on YouTube, but I wanted to combine computer vision, voice recognition, and robotics to build something more technically impressive — and genuinely useful to people.

H
Image Coming Soon
// 02 — Objectives

Build Checklist

Checked off as I go.

// Phase 1 — Robot Modeling
✓
Generate xacro file — 6 links, 6 joints, joint limits, inertia values Design
✓
Add camera link to end-effector with Gazebo Harmonic sensor plugin Design
✓
Load and verify all 6 DOFs in Gazebo Harmonic Software
✓
Visualise TF frames and joint states in RViz2 Software
// Phase 2 — Arm Control + IK
✓
Set up MoveIt 2 — run setup assistant, generate SRDF and controller configs Software
✓
Test IK in RViz2 — drag end-effector to arbitrary poses Software
✓
Write Python MoveIt client using MoveItPy API Software
✓
Test hardcoded pick-and-place motion in simulation Software
// Phase 3 — Vision (YOLOv8)
✓
Source a pre-labelled chess piece dataset from Roboflow Universe Vision
✓
Fine-tune YOLOv8n on dataset — achieved mAP50 of 0.982 across all 6 piece types Vision
✓
Validate model on real camera feed — verify live piece detection Vision
✓
Design overhead camera stand in SolidWorks — gantry style, sleeve joints, cable channel Design
✓
3D print and assemble camera stand — M4 bolted joints, locating lip for board alignment Hardware
✓
Map piece detections to board squares (a1–h8) using homography Vision
✓
Wrap YOLO board state tracker as a ROS 2 node publishing to /board_state Software
✓
Test board detection on simulated camera feed in Gazebo Vision
// Phase 4 — Voice Control
✓
Integrate OpenAI Whisper for real-time speech-to-text Voice
✓
Parse spoken move (e.g. "e2 to e4") into source and destination squares Voice
✓
Validate move legality against current board state from YOLO Software
✓
Wrap voice pipeline as a ROS 2 node publishing to /chess_move Software
// Phase 5 — Full Simulation Integration
✓
Compute hand-eye calibration — camera frame to robot base frame Software
✓
Map chess board square coordinates to robot workspace coordinates Software
✓
Build voice → vision → planning bridge node Software
✓
Implement full control loop — hear move, locate piece, pick, place, confirm Software
✓
Tune and stress-test in simulation — varied moves, captures, edge squares Software
// Phase 6 — Physical Build
✓
Design full 6DOF arm CAD in SolidWorks using xacro dimensions as reference Design
✓
Source components — servos, ESP32, Raspberry Pi 4, camera, PSU Hardware
✓
Breadboard prototype — verify ESP32 ↔ Pi UART comms and servo response Hardware
✓
3D print all arm links and assemble Hardware
// Phase 6b — Firmware
✓
Set up micro-ROS on ESP32 — agent, transport layer, build system Firmware
✓
Write PWM control firmware using ESP32 LEDC hardware timer API Firmware
✓
Subscribe to /joint_commands and drive all 6 servo outputs Firmware
✓
Validate firmware end-to-end — Pi publishes, ESP32 moves servos correctly Firmware
// Phase 7 — PCB Design & Fabrication
✓
Design schematic in Altium — ESP32, PCA9685, servo headers, UART, power input Hardware
✓
Route 2-layer PCB layout — wide servo power traces, separated ground planes Hardware
✓
Run DRC, review Gerbers, export Excellon drill file for JLCPCB Hardware
✓
Order boards from JLCPCB, solder components, bring-up test Hardware
✓
Swap breadboard prototype for finished PCB, re-run servo tests Hardware
// Phase 8 — Real-World Deployment
✓
Install ROS 2 Jazzy on Raspberry Pi, transfer and verify all packages Software
✓
Real hand-eye calibration using checkerboard + easy_handeye2 Software
✓
Retune YOLO under real lighting and board setup conditions Vision
✓
End-to-end real-world test — speak a move, arm executes it on a real board Deployment

Progress updated by Ramy

// 03 — Skills I'll Get Out of It

What I'm Learning / Mastering

// Mechanical Design
CAD & 3D Printing

Designing a full 6DOF arm in SolidWorks with manufacturable joints, linkages, and mounting points — then printing and iterating on real hardware.

SolidWorks3D PrintingDFM
// Electrical Design
PCB Design

Designing a custom PCB in Altium Designer to power and control the arm's actuators, handle signal routing, and interface with the ESP32 microcontroller.

AltiumSchematicsPCB Layout
// Software
ROS2 & MoveIt

Building a full ROS2 stack from URDF modelling to MoveIt motion planning, running inverse kinematics for real-time arm control driven by voice and vision inputs.

ROS2MoveItGazeboURDF
// Firmware
Embedded Firmware

Writing micro-ROS firmware for the ESP32 to subscribe to ROS 2 joint commands, drive servo PWM outputs via hardware timers, and communicate with the Raspberry Pi over UART.

ESP32micro-ROSC++PWM
// Computer Vision
Object Detection

Fine-tuning a YOLOv8 model on a chess piece dataset to achieve 98.2% mAP50, then deploying it to track full board state in real time and map piece positions to robot workspace coordinates.

YOLOv8PythonOpenCVRoboflow
// Voice & Integration
Voice Control Pipeline

Integrating OpenAI Whisper for real-time speech recognition, parsing spoken chess moves into board coordinates, and tying voice, vision, and motion planning into one autonomous system.

WhisperNLPROS2Integration