HAND
Personal Project — 2026 · In Progress

THE HAND

Status
In Progress
Year
2026
Type
Mechanical · Electrical · Software
Stack
ROS2 · YOLOv8 · Whisper · SolidWorks · Altium · ESP32
// 01 — What It Is

The Hand is an autonomous 6DOF robotic arm that lets people with motor disabilities play physical chess through voice control and computer vision — fully hands-free.

Many people living with conditions like ALS, muscular dystrophy, or severe arthritis love chess but physically can't manipulate the pieces. Existing solutions push them to digital-only play, removing the social experience of sitting across a real board from another person. The Hand bridges that gap — a voice command triggers the arm to pick up and move the correct piece on a real board, while YOLOv8 tracks the full board state in real time.

Built from scratch across mechanical design, PCB design, embedded firmware, computer vision, voice recognition, and ROS2 motion planning.

6DOF Robotic Arm
YOLOv8 Vision Model
ESP32 MCU
Whisper Voice Input
// 02 — Gallery
Custom trained chess detection CV model
Custom trained YOLOv8 chess detection model
Motion Planning (Inverse Kinematics) in Rviz2 with 6DOF robotic arm
01 / 01
// 03 — Build Checklist
// Phase 1 — Robot Modeling
✓
Generate xacro file — 6 links, 6 joints, joint limits, inertia valuesDesign
✓
Add camera link to end-effector with Gazebo Harmonic sensor pluginDesign
✓
Load and verify all 6 DOFs in Gazebo HarmonicSoftware
✓
Visualise TF frames and joint states in RViz2Software
// Phase 2 — Arm Control + IK
✓
Set up MoveIt 2 — run setup assistant, generate SRDF and controller configsSoftware
✓
Test IK in RViz2 — drag end-effector to arbitrary posesSoftware
✓
Write Python MoveIt client using MoveItPy APISoftware
✓
Test hardcoded pick-and-place motion in simulationSoftware
// Phase 3 — Vision (YOLOv8)
✓
Source a pre-labelled chess piece dataset from Roboflow UniverseVision
✓
Fine-tune YOLOv8n on dataset — achieved mAP50 of 0.982 across all 6 piece typesVision
✓
Validate model on real camera feed — verify live piece detectionVision
✓
Design overhead camera stand in SolidWorks — gantry style, sleeve joints, cable channelDesign
✓
3D print and assemble camera stand — M4 bolted joints, locating lip for board alignmentHardware
✓
Map piece detections to board squares (a1–h8) using homographyVision
✓
Wrap YOLO board state tracker as a ROS 2 node publishing to /board_stateSoftware
✓
Test board detection on simulated camera feed in GazeboVision
// Phase 4 — Voice Control
✓
Integrate OpenAI Whisper for real-time speech-to-textVoice
✓
Parse spoken move (e.g. "e2 to e4") into source and destination squaresVoice
✓
Validate move legality against current board state from YOLOSoftware
✓
Wrap voice pipeline as a ROS2 node publishing to /chess_moveSoftware
// Phase 5 — Full Simulation Integration
✓
Compute hand-eye calibration — camera frame to robot base frameSoftware
✓
Map chess board square coordinates to robot workspace coordinatesSoftware
✓
Build voice → vision → planning bridge nodeSoftware
✓
Implement full control loop — hear move, locate piece, pick, place, confirmSoftware
✓
Tune and stress-test in simulation — varied moves, captures, edge squaresSoftware
// Phase 6 — Physical Build
✓
Design full 6DOF arm CAD in SolidWorks using xacro dimensions as referenceDesign
✓
Source components — servos, ESP32, Raspberry Pi 4, camera, PSUHardware
✓
Breadboard prototype — verify ESP32 ↔ Pi UART comms and servo responseHardware
✓
3D print all arm links and assembleHardware
// Phase 6b — Firmware
✓
Set up micro-ROS on ESP32 — agent, transport layer, build systemFirmware
✓
Write PWM control firmware using ESP32 LEDC hardware timer APIFirmware
✓
Subscribe to /joint_commands and drive all 6 servo outputsFirmware
✓
Validate firmware end-to-end — Pi publishes, ESP32 moves servos correctlyFirmware
// Phase 7 — PCB Design & Fabrication
✓
Design schematic in Altium — ESP32, PCA9685, servo headers, UART, power inputHardware
✓
Route 2-layer PCB layout — wide servo power traces, separated ground planesHardware
✓
Run DRC, review Gerbers, export Excellon drill file for JLCPCBHardware
✓
Order boards from JLCPCB, solder components, bring-up testHardware
✓
Swap breadboard prototype for finished PCB, re-run servo testsHardware
// Phase 8 — Real-World Deployment
✓
Install ROS2 Jazzy on Raspberry Pi, transfer and verify all packagesSoftware
✓
Real hand-eye calibration using checkerboard + easy_handeye2Software
✓
Retune YOLO under real lighting and board setup conditionsVision
✓
End-to-end real-world test — speak a move, arm executes it on a real boardDeployment

Progress updated by Ramy

// 04 — What I'm Building / Learning
// Mechanical Design
CAD & 3D Printing

Designing a full 6DOF arm in SolidWorks with manufacturable joints, linkages, and mounting points — then printing and iterating on real hardware.

SolidWorks3D PrintingDFM
// Electrical Design
PCB Design

Custom PCB in Altium Designer to power and control the arm's actuators, handle signal routing, and interface with the ESP32 microcontroller.

AltiumSchematicsPCB Layout
// Software
ROS2 & MoveIt

Full ROS2 stack from URDF modelling to MoveIt motion planning, running inverse kinematics for real-time arm control driven by voice and vision inputs.

ROS2MoveItGazeboURDF
// Firmware
Embedded Firmware

micro-ROS firmware on ESP32 — subscribing to ROS2 joint commands, driving servo PWM via hardware timers, communicating with Raspberry Pi over UART.

ESP32micro-ROSC++PWM
// Computer Vision
Object Detection

Fine-tuned YOLOv8 model detecting chess pieces and tracking full board state in real time, mapping piece positions to robot workspace coordinates.

YOLOv8PythonOpenCVRoboflow
// Voice + Integration
Voice Control Pipeline

OpenAI Whisper for real-time speech recognition, parsing spoken chess moves into board coordinates, tying voice, vision, and motion planning into one system.

WhisperNLPROS2Integration