One the the primary goals of Heart Rate Touch is to produce the most accurate answers possible. Accuracy degrades with sensor noise, user motion, pulse irregularities, numerical errors, and, of course, software bugs. This blog post is a little bit technical, but shows how we quantified and improved our accuracy before we released 1.0. TL;DR It's pretty good.
The best way to gain confidence in software is to test, and with a little design you can make your software readily testable. Synthetic image testing is an approach you simulate image inputs from a known value. You can measure the actual output and compare it with the expected output. For Heart Rate Touch, it looks like this:
The green boxes above are test code where the red box is the actual production algorithm. (Incidentally, this is why you want to make your code modular. It lets you build interesting variations on your original work easily.) In the above example we generate images pulsing at 62 bpm. When we put it into the actual algorithm, it measures 63.1 bpm which means the error is 1.1 bpm. Why not zero? After some investigation we determined that error was coming from quantization error. Our algorithm does frequency analysis much like a graphic equalizer does on your stereo. The more bands you break your signal into the higher accuracy you get but the slower runs, so there is a speed accuracy tradeoff.
If we sweep over the entire operating range from 40 to 200 bpm, and simply pick the biggest frequency value, an interesting error pattern emerges.
A couple of things to notice here: (1) the error can get quite large, (2) the error is periodic and (3) sometimes, for certain rates, the error is zero. What causes this error? When the peak frequency is spread across multiple frequency bands, there will be error. We can quantify the error by adding up all of the errors, squaring them and then taking their square root. This is known as Root Mean Squared Error (RMSE) and ensures that positive and negative errors don't cancel each other out. In this case the RMS Error is 2.01 bpm.
A simple possible improvement for this is to fit data from multiple frequency bands. For example, if the strength of frequency band 10 and band 11 have equal values, we can conclude that the true peak is actually closer to 10.5. We can use even more bands and do a simple fit. Doing so reduces the error by almost half with a 1.18 RMS error.
Can we do better? To reduce error further we could double or triple the number of frequency bands we use. This would add delay the first measurement. Rather than incur this penalty, we came up a better solution. Using the rough peak frequency band as a robust initial estimate, HRT operates on the original signal to compute a very precise fit. The RMS Error drops to an astounding 0.05 bpm.
We are very happy with this result. (We have observed that some of our competitors have no where near this accuracy, being closer to the no-fit, naive solution.)
Synthetic tests are great, but they only test part of the system. How does it work in the real world? We compared Heart Rate Touch against the Polar FT7 with H1 cheststrap sensor and found good agreement. First we measured at a resting rate:
After playing an hour of Ultimate Frisbee, we then measured again. This time the heart rate was double the value but still tracking well.
We are quite pleased with the performance of 1.0 but that doesn't mean there isn't room for improvement. When the user moves or the pulse is irregular, it can sometimes result in an incorrect measurement. We are studying these problems and ways to address them. We look forward to being able to roll out these improvements in future releases and blog about them. Speaking of which, if you are interested in testing early beta versions, reach out to us at firstname.lastname@example.org. We can't guarantee that you will get a spot, but we would love to hear from you.