This post is also available in: heעברית (Hebrew)

There are many devices designed to try to track hand movements, using various technologies. Some watch your fingers using depth-sensing cameras or infrared sensors, some use motion-sensing gloves, and others use electromagnetic sensors on your fingertips. But almost all of them are a bit too bulky for practical use.

In a potential breakthrough in wearable sensing technology, researchers from Cornell and the University of Wisconsin, Madison, have designed a wrist-mounted device that continuously tracks the entire human hand in 3D.

With a bit more refining, the research team says that the FingerTrak system could have a range of uses. It could allow virtual reality players to track their own hand movements in-game, it could make for remote-controlled robots that directly mimic a human operator’s motions and applied for human-robot interaction, an aspect that could be implemented in the military field. Specialized robots under human teleoperation have proven successful in hazardous environments such as in policing or security missions.

The technology could also help translate sign language into text or speech, or help monitor health issues that affect motor skills.

The new wearable system uses thermal sensors to accurately predict hand positions. It is basically a bracelet adorned with four small thermal cameras, each about the size of a pea. From their respective positions, the cameras snap images of the contours of the wearer’s wrist. A specially-designed algorithm accurately reconstructs the entire hand, including the positions of each finger.

According to, the FingerTrak system uses machine learning to predict the position of 20 finger joints, based on the wrist contours. These hand poses can then be recreated on a virtual model, or even a robot hand. In tests, the device was able to accurately reproduce actions like opening a book, writing with a pen, drinking, and using a phone.

The research was published in Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies.