It's really amazing how much the role of technology has increased in the past 10 years. Nowadays, technology promises to bring an answer to many of our questions. If you ask yourself why, the answer is pretty obvious: we live in an age where we have to deal with and work with a huge quantity of data and the results must be fast and very accurate.
The medical field is not an exception. Doctors and researchers start turning to technology for solutions in improving or replacing existing procedures. From blood analysis to brain surgery, everything is supervised and guided by software products or hardware devices specially created for this purpose. The material and intellectual investment required for the development of such solutions is tremendous and it's not easily accessible to every programmer.
But luckily, despite many techies not paying much attention to it, the gadget market has been developing a lot and it started offering many high potential, relatively cheap solutions for many problems. My statement regarding the low-key nature of this new market is not a random one: almost all the progress registered in this domain has been focused on developing games and gaming applications or on making these games as accessible and user-friendly as possible. Developers have created all sorts of playful sensors, one more interesting and more captivating than the last, but all having the same main purpose: to make the experience of the user as pleasant as possible, whether this experience takes places in a virtual environment or a real one. I will not go through all the gadgets that are out there; instead, I will focus on Microsoft Kinect.
Kinect is a device that comes together with the Xbox (360 or One) and it is actually a video camera, complete with a sensor that is able to detect body movement. Its purpose is obviously oriented towards the gaming industry.
But how does it actually work? The sensor from the camera creates a depth map with the help of the infra-red rays; these rays help determine the distance from the camera to the object in front of it.
The resulting depth map helps identify the 20 main points of the skeleton which will be used to reproduce the individual movements of the body.
This sensor is able to detect motion starting from 0.7 meters and going as far as 6 meters, under the condition that the light in the room is a natural one (although this aspect is no longer a problem for Kinect One, as Microsoft has stated).
Moreover, the sensor is able to recognize and map approximately 100 points at face level, which is extremely important because there are very few devices that are able to do this and that come at such a convenient price.
If you are wondering how we managed to bring Microsoft Kinect and medicine together, the following story will shed light on the whole "affair". My colleagues, Rares Urdea and Bogdan Pop, and I, coordinated by our teacher, Dan Mircea Suciu, managed to bridge this gap together.
More than a year ago, we decided to go for the win in a contest called Imagine Cup. This is one of the most prestigious contests from the field of software development, organized solely for high-school and college students. The only problem was that, even after hours and hours of thinking and brainstorming, we still couldn't find an idea that would take us to the finals. Our teacher however, who had already won this competition in the past, came up with the idea of connecting Kinect and the medical field.
This is how sMilestone was born. sMilestone is an application meant to aid patients suffering from different types of facial paralysis regain the control of their facial muscles. More precisely, the games from the app can replace the often uncomfortable exercises that the patients go through during their recovery therapy sessions. The games manage to capture the patients' attention much more easily and in a more fun way. Moreover, the app makes it possible for physicians and therapists to measure a patient's progress more easily and to obtain more accurate data and statistics. Another advantage of this solution is that the therapist can assist more patients at the same time and, most importantly, this is a non-invasive procedure.
By using the sensor's ability to detect the main facial points, we've created algorithms corresponding to the exercises performed by the patient during a therapy session.
To provide some examples, the game Snake from the sMilestone application allows its user to control the snake through 2 simple lip moves: a smile to go right and an open mouth to go left. This way, instead of asking the patient to simply perform these actions in front of a mirror, exercising becomes a lot more fun and captivating since timing and keeping a score are involved. The algorithm behind the detection of these moves is based on calculating the distance between two pairs of points: 60-64 (the corners of the mouth) for the smile, 62-66 (upper and lower lips) for opening the mouth. The moment the game begins, the algorithm determines the neutral position by calculating the initial position between the two sets of points. After that, depending on how much the pairs of points distance themselves from their initial position (this distance varies from patient to patient given that each patient is in a different stage of recovery), the algorithm "decides" whether the expression produced by the patient is a smile or an open mouth. Of course, this algorithm is also responsible for keeping track of the scores and records.
For exercises of this type, the sensor and the SDK from Kinect offer us everything we need. But the biggest challenge my colleagues and I faced was creating an algorithm that would be able to detect the movement of the tongue. This was almost unconceivable, especially since our solution didn't involve sticking sensors on the patient's skin.
After several attempts, we managed to create an optimal and unexpectedly efficient algorithm. Setting a limit for the surface that must be processed, more precisely the area of the mouth and using the depth map we managed to isolate the tip of the tongue. This way, when the patient sticks out his tongue, the sensors isolate the tip of the tongue and we use it to detect its moves. Isolating the tip of the tongue is possible due to the depth map, which is created according to the distance of the objects relative to Kinect, calculated in millimeters. When the patient sticks out his tongue, its tip is the closest point to the sensor and this is how the algorithm can determine that we are dealing with the movement of the tongue (see image).
After this, the possibilities became almost unlimited. After creating Dots, a game that allows the patient to duck the balls chaotically directed at him by moving his ball using the tip of his tongue, we made some changes so that sMilestone allows its user to control the app's menu with the help of his tongue. This is the tweak that allowed us to get to the finals.
As you've read so far, Kinect's SDK comes with a full package of libraries for capturing the movements of the body or face. It also includes the support for voice recognition, which I must confess is pretty detailed and a more thorough discussion about it would require another article. I can also tell you that, for the time being, its biggest support is in English given that the main beneficiary is the gaming industry. But it is the perfect candidate for facilitating the therapy of the patients who are in a more advanced stage of the recovery process (to be more precise, it is for the patients who can already practice pronouncing sounds and words).
To sum things up, I hope I've made you curious about this device and I wish that you would also try to "toy" with the functions of Kinect so that, in the future, you too could try and create something as impactful and as helpful. All I can tell you is that it is extremely satisfying to know that you can help people by doing what you love most.
by Peter Lawrey
by Răzvan Costa
