Personal project — gesture control

IntuiMotion

Touch-free mouse, macro, and media control for the original Leap Motion Controller. An open, still hand becomes a pointer, a pinch becomes a mouse button, and a swipe changes the track, all riding on Ultraleap's Gemini hand tracking stack.

Gesture pipeline: confirmed on live hardware Calibration solvers: unit-tested, synthetic only Camera screen detection: in progress View source on GitHub

What it is

A small pointer replacement: wave your hand in the air near a Leap Motion sensor and it moves your cursor, clicks, drags, scrolls volume, and skips tracks, no touchpad, mouse, or keyboard involved.

Ultraleap's Hand Tracking Service does the actual computer vision, turning raw sensor frames into per-hand skeletal data. IntuiMotion sits above that: it watches the tracked hand for a small vocabulary of deliberate gestures, and maps each one to a mouse action, a media key, or a custom macro (a keystroke combo or a shell command) defined in a plain config file, no code changes needed to add a new one.

No live demo on this page. This is a write-up of the project as it stands, not a control panel; running it for real requires the physical Leap Motion sensor and Ultraleap's tracking service installed on the machine in front of you.

Gesture vocabulary

The default gesture-to-action mapping. Idle-mode gestures fire while you're not pointing at anything; pointer-mode gestures fire once an open hand has engaged the cursor.

Gesture Mode Trigger Default action
palm_engage Idle → Pointer Open hand held still for about 0.4s Enters pointer mode
pinch Idle Thumb+index pinch Play / pause
left_press / left_release Pointer Thumb+index pinch start / end Left mouse button down / up
right_press / right_release Pointer Thumb+middle pinch start / end Right mouse button down / up
fist_exit Pointer → Idle Fist held for about 0.15s Exits pointer mode
swipe_up / swipe_down Idle Fast vertical hand motion Volume up / down
swipe_right / swipe_left Idle Fast horizontal hand motion Next / previous track

Mouse buttons mirror pinch state directly instead of firing a one-shot click: press on pinch-start, release on pinch-end. A quick pinch reads as a click, a held pinch reads as a drag, the same as a physical mouse button, with no separate tap-versus-hold timing logic required.

Idle-only gestures are briefly suppressed for a short grace window right after fist_exit, so a hand relaxing out of the fist shape doesn't misfire a volume change or play/pause. palm_engage is exempt from that window; an open, still hand can re-engage pointer mode at any time.

Right-click detection uses thumb-to-middle-finger distance rather than the tracking library's built-in pinch metric, which is thumb-index only. Whether an ordinary thumb-index pinch can also cross that threshold when the fingers are held close together hasn't been confirmed against a real hand yet.

Architecture

Frame data flows one direction, from sensor to OS action:

  1. Leap Motion Controller (USB)
    the sensor itself
  2. Ultraleap Hand Tracking Service
    system daemon doing the actual hand tracking
  3. LeapC Python bindings
    leap.Connection / leap.Listener
  4. TrackingListener
    per-hand frame callback
  5. GestureInterpreter
    raw hand data → discrete gesture events + mode (idle / pointer)
  6. ActionDispatcher
    gesture name → config-mapped action
  7. Mouse / media / macro actions
    pynput mouse, media keys, keystrokes, or a shell command

Two things skip that pipeline on purpose. Pointer-mode cursor movement is continuous per-frame motion, not a discrete triggered event, so it's wired directly rather than routed through the dispatcher. Mouse-button press and release work the same way, since they need paired press/release calls tied to the physical timing of the pinch, not a single fire-and-forget action.

Screen calibration

Two approaches to teaching the app where your screen physically is. They're not two attempts at the same problem, they solve different versions of it.

Newer, recommended for monitor/TV

Touch-based edge tracing

Hold both pointer fingers out at the screen's top corners until they lock in place, then trace both fingers down along the screen's real physical edges to the bottom corners, holding still again to lock the endpoint.

Because the fingers stay on or near the actual screen surface the whole time, each corner gets measured as a real 3D position directly, with no separately-known screen size required. The traced path is smoothed against the locked points using PCA, for robustness against ordinary hand tremor.

Ten unit tests, including recovery from a synthetic noisy trace and a regression guard against the end-lock firing immediately after the start-lock, before any real tracing motion has happened.

Needs real-hardware validation

Ray-walking (corner-pointing)

Point at a screen corner, walk backward while staying pointed at it, close a fist to solidify that corner, and repeat for all four corners in a fixed order. Once all four are solidified, a geometry solver derives the screen's actual corner positions.

This technique needs the screen's real physical width and height as input. Testing confirmed that ray geometry alone, right angles, equal sides, a shared plane, with no absolute scale, doesn't uniquely determine the rectangle: two different rectangles can satisfy the same relative-shape constraints from the same four rays. A known size is what actually pins down scale and position.

Sixteen unit tests, including two that check the solver against known synthetic 3D ground-truth rectangles rather than just mocked gesture logic. Not yet run against a real calibration session; every timing and distance threshold is an untuned starting point.

Edge tracing needs something physical to trace against, so ray-walking stays in the codebase for scenarios without a touchable surface, most plausibly a future augmented-reality mode, where there's nothing physical to run a finger along.

Camera-based screen detection

In progress, not wired in yet

An OpenCV-based screen finder, absorbed from a standalone computer-vision prototype once that project's camera and detection logic moved onto the same machine as the rest of IntuiMotion.

The screen detector looks for the largest 4-sided bright rectangular region in a webcam frame using plain OpenCV contour detection and polygon approximation, no trained model, and returns its four corners in the same canonical order (top-left, top-right, bottom-right, bottom-left) the calibration modules already use.

It isn't wired into the live gesture pipeline yet, and hasn't been run against a real camera; validation so far is synthetic frames and mocked capture calls only. Turning “a camera can find a screen's corners in pixel space” into “point at that real screen and control it like a touchscreen” needs a coordinate-fusion step, a homography from the camera's pixel-space corners onto the same 3D plane the hand-tracking calibration already solves against, that hasn't been designed yet.

Status & known gaps

What's confirmed against real hardware, what's tested but not field-proven, and what's deliberately left untuned for later.

Confirmed

Core gesture pipeline

Connects to a real sensor, tracks a real hand, and dispatches real mouse and media actions end to end.

Tested, not field-proven

Calibration & gesture logic

The state machine, config loading, and both calibration solvers are covered with fake hand data and synthetic geometry, not yet run against a real calibration session.

In progress

Camera screen detection

Detection logic exists and is unit-tested against synthetic frames; not yet wired into the live pipeline or run against a real camera.

Known gaps

  • Cursor mapping bounds are untuned guesses. Expect jitter or dead zones until they're adjusted against a given sensor's placement and a real hand's actual range of motion.
  • Pinch, grab, swipe, and dwell thresholds are untuned starting points, set from limited live debugging rather than systematic tuning. The fist-exit dwell requirement exists because a firm pinch can briefly spike the grab signal past its own threshold, so the fist pose has to hold for a short window instead of firing on a single frame.
  • No smoothing or deadzone on cursor movement yet. Raw palm position maps straight to screen pixels every frame.
  • Two hands in pointer mode at once will fight over the cursor. Each hand tracks its own mode independently, but there's only one OS cursor, so whichever hand's frame gets processed last each tick wins. Not an issue for the intended use case of one hand on the pointer while the other performs gestures.
  • The right-click threshold hasn't been checked against an ordinary pinch, to confirm thumb-to-middle-finger distance and thumb-to-index-finger distance can't be confused when the fingers are all held close together.
  • Nothing in continuous integration exercises the compiled tracking bindings themselves, since that requires the physical sensor; only the gesture-logic state machine and config loading are covered by the automated test suite.