poseHook

This is an app that continuously detects the body parts of a single person in the frames seen by your device's camera. It transmits the detected positions to a host in form of OSC messages. OSC is a UDP based protocol understood by many music or visual art software, e.g. TouchDesigner, Max for Live, Resolume Arena, etc.

You can download Ableton Live sets demonstrating OSC conversion to MIDI, and much more:

https://bitbucket.org/constanze/posehook/

Detected are: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. It is possible to configure the OSC address names, the minimum confidence and IP / port of the receiver. It doesn't support detection of multiple persons.

poseHook uses TensorFlow PoseNet lite for the detection. The OSC messages are sent as fast as they are detected. A frame rate of > 20fps is possible with a S10-like phone.

On host side, you can receive OSC messages with various kinds of musical or creative coding software. Note that the data is quite noisy, so you have to take that into account when creating an effect. Increasing the minimum detection confidence reduces the noise, but also filters some data.

Questions how to use poseHook? Or would you like to share your idea for improvement? Get in contact with us on twitter: @holllyhook (trippple L).

The person detection works to 100% offline, the data is neither saved nor shared, besides of the intended OSC usage.

poseHook is open source: https://bitbucket.org/constanze/posehook/

NB: this is poseHook v1. The next major release will integrate MediaPipe model instead of PoseNet, which is an incompatible change in output format. Sorry for that. The v1 apk will still be provided for sideloading. A preview of the v2 is already available if you opt in for beta testing.