Camera + MIDI experiment 003 / Gesture Orchestra / Live

Interactive music instrument

Gesture
Orchestra

A camera-driven performance space where your hands behave like a live controller. Notes, drums, filters, and MIDI data all shift with the gesture you make.

Best used on Desktop / HTTPS
  • Open palm wakes the engine
  • Pointing maps your hand position to notes
  • Pinch shapes filter, energy, and MIDI CC
  • Peace sign swaps musical scale
Hand tracking / Browser synth / MIDI output / Gesture control / Live performance

Live stage

Play with your hands, not a timeline.

Start audio and camera, then use a single hand to perform. The built-in synth plays in-browser, and an external MIDI output can receive note and control data too.

Stage

Tracking window

Tracking offline No gesture

Waiting to start.

Gesture map

Five moves, one instrument.

The page reads broad hand poses instead of tiny finger choreography, so it stays readable in a normal webcam setup.

Open palm

Wakes the engine and lets the rhythm move again.

Fist

Stops notes, freezes playback, and sends all notes off to MIDI.

Point

Maps left-to-right position to notes in the current scale.

Pinch

Changes filter brightness, output energy, and MIDI control messages.

Peace sign

Cycles through scales so you can shift mood mid-performance.

Tech stack

MediaPipe sees the hand.
Tone.js turns it into sound.

The gesture layer runs in the browser with no backend. Hand landmarks drive note selection, drum accents, filter movement, and Web MIDI messages for external gear.