A while ago, with a little urging from friends, I decided to change how the wands identify gestures. Right now, the wands recognize eight basic positions:
- Pointing straight ahead,
- Pointing straight ahead but rotated 90º clockwise,
- Pointing straight ahead but rotated 90º counterclockwise,
- Pointing directly at the ground,
- Pointing downward at roughly 45º,
- Pointing straight upward,
- Pointing up at roughly 45º, and
- Pointed anywhere behind you.
When the wand changes from one of these positions to another it makes a note of it. It remembers the last 20 positions it’s been in. Each time it changes position, it checks to see if the most recent position sequence matches a spell’s sequence. For example, say a spell starts straight ahead (1), turns 90º CCW (3), then goes straight ahead (1), then goes 90º CW (2), then goes straight ahead again (1). Every time the wand changes position, one of the sequences it’s looking for in its position memory is “1, 2, 1, 3, 1.” Note that this is in reverse order because that’s how the wand’s memory works.
This system is all good and fine, but it’s not very fine-grained. When people pick up the wand, the first thing they do is make swishy wand-gestures in front of them. It would be great if the wands could actually recognize that kind of gesture.
That’s why I want to use AI to teach the wands to recognize more complex and subtle gestures. Specifically, I want to use Google’s TensorFlow. Why? It’s the right combination of price (free), complexity (moderate), and it should be more than capable of doing what I need it to do.
More specifically, I want to use TensorFlow Lite, which is designed specifically to run on microprocessors with limited resources (like an ESP32!). I was taking a break from setting up my computer with Espressif’s ESP32 development environment – a prerequisite to installing the needed TensorFlow Lite software – when I found a GitHub blog called Eloquent Arduino. In a post titled Easy TinyML on ESP32 and Arduino, the author shares their experience with using TinyML to deploy TensorFlow Lite models on microcontrollers. They went through all the hassle of getting it set up, created a library for the Arduino IDE, shared it with the world, and wrote up their entire process along with examples of how to use it.
I launched the Arduino IDE, used the Library Manager to download and install the Eloquent TinyML library, opened the SineExample sketch (from File >>> Examples >>> Examples from custom libraries >>> Sine Example), compiled and uploaded it to an ESP32, and it worked like a charm.
In about 10 minutes, I had TensorFlow Lite running on an ESP32. It’s like magic. The Eloquent TinyML library likely saved me hours. I’m quite chuffed, really. Now I can focus on training the wand, getting that data formatted for Tensor Flow, training a net/model, getting the results TensorFlow Lite-ready, and dropping that on the wand.
It’s gonna be great!