Cosmic Hand Control
LiveOverview
Cosmic Hand Control is a browser-based 3D interaction playground where webcam hand gestures control floating iridescent blocks. It combines React 19, TypeScript, Three.js, React Three Fiber, Drei, and MediaPipe Hands to turn a normal webcam feed into a real-time gesture controller.
The app tracks 21 MediaPipe hand landmarks, classifies gestures, and maps palm position into the Three.js scene. Open palm selects blocks, pinch grabs and moves them with spring-like physics, fist destroys a selected block with particles, and thumbs up duplicates the selected object. The scene itself uses floating cubes, collision response, stars, fog, lighting, and a mirrored webcam overlay to make the interaction understandable while it runs.
Architecture
Challenges & Solutions
High-frequency hand data can cause React render churn
The implementation keeps live hand and physics data in refs, then updates the Three.js scene inside the render loop instead of pushing every frame through React state.
2D webcam landmarks need to feel natural inside a 3D scene
Palm coordinates are projected through the camera into scene space, then used to hover, select, and spring-grab nearby blocks.
Gesture actions need clear feedback
The app pairs selected/hovered block styling with status updates, duplication cooldowns, and particle explosions so each gesture has visible cause and effect.
Gesture-driven 3D controls
Tracking
21 points
MediaPipe Hands detects hand landmarks from a mirrored webcam feed in real time.
Scene
15 blocks
Floating iridescent cubes run inside a React Three Fiber scene with lighting and stars.
Gestures
4 modes
Open palm selects, pinch grabs, fist destroys, and thumbs up duplicates blocks.
Features
- Real-time webcam hand tracking with MediaPipe Hands
- Open palm selection, pinch grabbing, fist explosion, and thumbs-up duplication
- React Three Fiber scene with Three.js lighting, stars, fog, and environment preset
- Floating iridescent blocks with collision response, drag, restitution, and spring-based grabbing
- Particle explosion effects when selected blocks are destroyed
- Camera permission handling and mirrored webcam preview overlay
- Performance-focused refs for high-frequency hand and physics updates
