Created Friday 17 November 2023
Did you know that with APDS9960 it is possible to
- Measure both Speed and Angle of hand
- Recognize Rotary Knob / Airwheel gestures
- Perform fine finger motion tracking
Unfortunately, all existing libraries will make you look like a clunky clown gesturing:
Tired of aligning your hands like a robot to make those stupid libraries recognize at least something?
It's time to change the approach, change the algorithm!
How to create robust signal processing algorithm for APDS9960 and similar sensors? Keep reading, and you will learn how..
To make a really good algorithm you need to follow these steps:
- Make a firmware which allows you to stream sensor data to computer
- Record sensor data to files
- Develop algorithm using computer by constantly replaying those files as if sensor is connected
Only this way you can create a great algorithm.
If something is not detected, then find out why. Improve your algorithm. Use neural networks, anything that can help!
Then you can transfer your algorithm to microcontroller.
Success rule number one - work with recorded data!
Have you seen any APDS-9960 based design which can reliably detect gestures? Nobody uses them for serious tasks. You are going to learn how to create APDS9960 algorithm that works really good. Learn from my design steps.
First - I recorded typical movements (slow and fast). Here is the DATASET:
- g-up-down.raw - up/down movements
- g-left-right.raw - left/right movements
- g-ul-dr.raw - diagonal movements from upper-left to down-right corner
- g-ur-dl.raw - diagonal movements from upper-right to down-left corner
- g-cw.raw - clockwise forefinger rotary gesture
- g-ccw.raw - counterclockwise forefinger rotary gesture
note: this dataset is already passed to smoothing algorithm on stm32, more details will come soon
You can find find these files in the attachments..
I will describe STM32 gesture signal recorder firmware creation in other article. Generally, you want to record data at highest possible frame rate without any interruptions. It is very similar to computer vision algorithms. At this stage, the more data you have, the better.
Second - I started to develop an algorithm.
Note that it is an iterative process. For APDS9960 Pro project, I choose JavaScript for algorithm prototyping. I've created a HTML file and used javascript functions to load RAW gesture files, draw wavforms and test various algorithms. I made around 9 iterations,
- apds-v1.html - first attempt, recognizes only LEFT/RIGHT gestures (uploaded on 2023-11-17)
- apds-v2.html - a little better
- apds-v3.html - better
- apds-v4.html - better and better
- apds-v5.html - starts to look interesting
- apds-v6.html - very good results
- apds-v7.html - even better results
- apds-v8.html - good working algorithm
- apds-v9.html - the best, but I know how to make it even better!
These files will be released gradually, along with youtube videos. And don't forget to become a Premium Member to support my work!
Third step - transfer algorithm to STM32
Now, when JavaScript algorithm recognizes 99% of recorded gestures, algorithm is ready for real-time measurements on stm32. This will be covered in next articles.
apds-v1.html - beginning of APDS-9960 Pro
Algorithm uses simple state machine that analyzes waveform derivatives (see attachments).
Recognition principle (apds-v1):
- calculate waveform derivatives (first/second)
- analyeze derivative sign
- use state machine to recognize left/right gestures