Personal project — USB driver & hand tracking

OpenMotion

A fully open-source Rust/libusb driver and hand-tracking stack for the original Leap Motion Controller. No leapd, no closed SDK, no dead vendor services—just the hardware, reverse-engineered USB control writes, and Monado’s openly-licensed Mercury hand tracker.

Device bring-up & streaming: confirmed on real hardware 2D + 3D hand tracking: validated (depth matches ruler) Monado Mercury engine path: in progress View source on GitHub

What it is

The Leap Motion Controller (USB f182:0003) is a small stereo-IR sensor that shipped in 2013 for hand tracking. Its maker, Ultraleap, abandoned the original hardware: the SDK repository is dead, the tracking service is unmaintained, and the device is otherwise e-waste.

OpenMotion replaces the entire proprietary stack. A Rust driver handles USB bring-up and stereo IR streaming, a Python pipeline runs Monado’s openly-licensed Mercury ONNX models for 21-joint hand skeletons, and the whole thing feeds gesture input to IntuiMotion—the gesture-control layer that turns tracked hand poses into mouse actions, media keys, and macros.

Named in tribute to the 2013 OpenLeap initiative, which set out to build an open driver for this hardware and stopped at raw image access. OpenMotion is an independent implementation (no shared code) that completes the mission: device bring-up, calibrated stereo capture, 21-joint hand skeletons, and 3D triangulation in millimeters—all open.

Device bring-up

The controller is a UVC-shaped device that does not work with standard UVC drivers—it boots inert and needs a vendor bring-up sequence. There is no crypto handshake: it just needs the right control writes.

  1. Claim interfaces
    Detach uvcvideo from interface 0 (VideoControl, owns the vendor extension units) and interface 1 (VideoStreaming, owns bulk IN endpoint 0x83)
  2. Replay 358 UVC control writes
    The full bring-up sequence (bringup_seq.txt)—sensor config, LED driver init, UVC probe/commit—decoded from a usbmon capture of the closed leapd talking to the device
  3. IR LEDs strobe, sensor streams
    After bring-up, the device is live: three IR LEDs illuminate the scene and the stereo sensor begins delivering frames over the bulk endpoint

Everything above was decoded from USB traffic. The protocol knowledge was obtained by observing the device’s own USB traffic with usbmon, not from any SDK source code or documentation. No Ultraleap code, blobs, or SDK output is included or required. See LICENSING.md in the repository for the full clean-room posture.

Streaming architecture

Once the device is brought up, a robust capture pipeline keeps frames flowing at USB 2.0 saturation without tearing or data loss.

Frame format

Bulk endpoint 0x83 delivers UVC payloads: 16,380 bytes per transfer (12-byte header + 16,368 pixel bytes). Frames are delimited by EOF bits, with FID-toggle recovery when an EOF payload is lost.

One frame = 307,200 bytes: 640×240 per eye, the two eyes interleaved column-by-column into 1280×240, 8-bit grayscale, at roughly 112 fps.

Drain thread + mailbox

A dedicated drain thread running at SCHED_FIFO real-time priority does nothing but service the endpoint into a one-slot mailbox. A writer thread ships the latest frame to stdout and drops frames if the consumer is slow.

The endpoint is never blocked on a pipe—torn frames went from >50% to roughly 1% with this design.

Stream format on stdout: "LFRM" magic + u32 LE frame counter + 307,200 raw bytes per frame. Stderr gets a diagnostic line every 128 frames (brightness, exposure, LED mask, and frame-integrity counters).

Device controls

Runtime control over the sensor’s exposure, IR illumination, and on-device calibration data, all via standard UVC SET_CUR / GET_CUR requests.

Working

Exposure

Camera Terminal (unit 2) selector 0x0b, 4-byte µs little-endian. Effective range roughly 100–1000 µs; values above 1000 are near-saturation.

Working

IR LEDs

Processing Unit (unit 5) selector 0x03. Three independently addressable LEDs (left, center, right) via a bitmask.

Working

Calibration reader

156 bytes via “property knocking”—write an address to PU sharpness, read a byte back from saturation. Contains per-eye focal/distortion coefficients and the 40.01 mm stereo baseline.

Tracking pipeline

Per eye, per frame—the Python reference pipeline runs Monado’s openly-licensed Mercury ONNX models to go from raw IR pixels to 21-joint hand skeletons in millimeters.

  1. Detect
    grayscale_detection_160x160.onnx on the full frame; 2 hand slots, confidence threshold gates noise
  2. Crop & orient
    Square box around each detection, resized to 128×128; the keypoint net knows one handedness, so the other hand’s crop is mirrored and auto-calibrated per track
  3. Keypoints
    grayscale_keypoint_jan18.onnx: 21 heatmaps (22×22), argmax + center-of-mass refine; previous frame’s joints feed back as a temporal prior
  4. Coast
    The detector only acquires (it dies past roughly 150 mm). Tracked hands keep going by cropping around the previous joints—survives to 300 mm+
  5. Smooth
    Per-joint One-Euro filter, confidence-weighted; still-hand jitter drops to roughly 1.5 px
  6. Triangulate
    Stereo-rectified joints from both eyes → per-joint disparity → depth (Z = fx·B/d, baseline from on-device calibration) → 21 joints in millimeters, validated against a ruler

This pipeline is the reference/validation oracle, not the product. The production engine is Monado’s Mercury hand tracker, fed by OpenMotion’s leap_open Monado driver (a C ABI exported from the Rust driver as libopenleap.a). The Python pipeline stays for offline analysis and as the ground truth that Mercury’s output is validated against.

Status & known gaps

What’s confirmed against real hardware, what’s validated but pending integration, and what’s next.

Confirmed

USB driver

Device bring-up, stereo streaming at roughly 112 fps, exposure/LED control, and calibration reader all working on real hardware.

Validated

Python tracking pipeline

2D keypoints + 3D triangulation match a physical ruler. Pinch noise roughly 1 px after smoothing. Now the reference oracle, not the product path.

In progress

Monado Mercury engine

The leap_open Monado driver feeds Mercury for full 3D skeletons. Next: decide niri integration API, tune Mercury jitter, upstream the driver.

What’s next

  • niri integration API. Choose between a LeapC compatibility shim and a native joint-set protocol for feeding gesture signals into the niri Wayland compositor.
  • Mercury jitter tuning. The Python pipeline’s One-Euro filter is well-tuned; Mercury’s equivalent needs the same pass against real-hand data.
  • Gesture signal extraction. Pinch, grab, and swipe with hysteresis, bridging OpenMotion’s skeleton output to IntuiMotion’s gesture interpreter.
  • Upstream the leap_open driver to the main Monado repository.