As mobile and wearable devices continue to grow smaller, it gets tougher for people to interact with screens the size of a matchbook. That could, however, may soon change thanks to a new sonar technology developed by a team of University of Washington engineers. Their latest project, FingerIO, will enable you to interact with your gadgets by simply writing or gesturing on any nearby surface, whether that’s a tabletop, a sheet of paper or even in mid-air.
“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” explains Rajalakshmi Nandakumar, a UW doctoral student in computer science and engineering. “I don’t need to instrument my fingers with any other sensors — I just use my finger to write something on a desk or any other surface and the device can track it with high resolution.”
FingerIO works by tracking fine-grained finger movements and turning a smartphone or smartwatch into an active sonar system using the device’s own microphones and speakers to emit an inaudible sound wave. That signal then bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space. Since sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside their front pocket or a watch hidden under the sleeve of their jacket.
To accomplish this feat, the researchers employed a type of signal typically used in wireless communication called Orthogonal Frequency Division Multiplexing, and demonstrated that it was capable of achieving high-resolution finger tracking with sound. Their algorithms leverage the properties of OFDM signals to track phase changes in the echoes and correct for any errors in the finger location to achieve sub-centimeter finger tracking.
According to its creators, using sound waves to track finger motion offers several advantages over cameras — which don’t work without line-of-sight when the device is hidden by fabric or another obstructions — and other technologies like radar that require both custom sensor hardware and greater computing power.
To test out their new concept, the UW crew devised a FingerIO prototype Android app and downloaded it to an off-the-shelf Samsung Galaxy S4 smartphone and a smartwatch customized with two microphones which are needed to track finger motion in two dimensions. (Today’s smartwatches typically only have one which can be used to track a finger in one dimension.) The average difference between the drawings and the FingerIO tracings was 0.8cm for the smartphone and 1.2cm for the wearable.
The researchers will be presenting their paper in May at the Association for Computing Machinery’s CHI 2016 conference in San Jose, California. Intrigued? Those wishing to learn more can see FingerIO in action below or head over to its website here.
[Images: Dennis Wise / University of Washington]