How to Fly a Drone Exclusively with Your Eye Movement?
This post is also available in: 
עברית (Hebrew)
Controlling drones with only the eye movement is becoming a reality. Roboticists from the University of Pennsylvania, U.S. Army Research Laboratory, and New York University have developed a pair of lightweight gaze-tracking glasses and a small computing unit, enabling a small drone to fly wherever you look.
While we’ve seen gaze-controlled drones before, what’s new here is that the system is self-contained, and doesn’t rely on external sensors, which have been required to make a control system user-relative instead of drone-relative. For example, when you’re controlling a drone with a traditional remote, that’s drone-relative: You tell the drone to go left, and it goes to its left, irrespective of where you are, meaning that from your perspective it may go right, or forwards, or backwards, depending on its orientation relative to you, according to spectrum.ieee.org. The trick is being able to localize the drone relative to the user without having to invest in a motion-capture system, or even rely on GPS.
The Tobii Pro Glasses 2 is a lightweight, noninvasive, wearable eye-tracking system that also includes an IMU and an HD camera. The glasses don’t have a ton of processing power onboard, so they’re hooked up to a portable NVIDIA Jetson TX2 CPU and GPU.
With the glasses on, the user just has to look at the drone, and the camera on the glasses will detect it, using a deep neural network, and then calculate how far away it is based on its apparent size.
Along with head orientation data from the IMU (with some additional help from the camera), this allows the system to estimate where the drone is relative to the user.
Then it’s just a matter of fixating on somewhere else with your gaze, having the glasses translate where your eyes are looking into a vector for the drone, and then sending a command to the drone to fly there. There is one other hard part, which is turning where your eyes are looking into a 3D point in space rather than a 2D one.
The researchers are hoping that eventually, their system will enable people with very little drone experience to safely and effectively fly drones in situations where finding a dedicated drone pilot might not be realistic. It could even be used to allow one person to control a bunch of different drones simultaneously.
