VR Gesture Detection Training System

Summary

A custom ML model that recognizes 50+ unique VR gestures in real time from headset and controller motion. Trained in TensorFlow, exported to ONNX, run inside Unity. Built in under two months.

Problem

VR controllers can replicate buttons, but the headset and controllers as a whole carry rich physical motion data. Gesture-based interaction feels more natural than menus and buttons. The challenge is converting continuous 6DOF motion across multiple inputs into a fixed-size representation a model can classify.

Architecture

Encode physical gestures into image representations, treating motion sequences as 2D pixel arrays. Train a CNN in TensorFlow Python on hundreds of recorded instances per gesture. Export to ONNX. Run ONNX inference inside Unity C# at runtime against the same encoding pipeline used during training.

Outcomes

• 50+ unique gestures classified in real time
• Full system completed in under two months
• Used as core technology in Mystic Sign, a VR spell-casting game
• Cross-platform VR support across Quest and SteamVR

Hardest bits

• Encoding continuous 6DOF motion across multiple inputs into a stable 2D image representation
• Keeping the encoding pipeline identical between Python training and Unity inference paths
• Real-time classification at VR frame rates without dropping frames

Stack