Audio Extracted from Still Images and Silent Videos- Uses and Dangers

This post is also available in: עברית (Hebrew)

Academic researchers from four US universities used artificial intelligence and machine learning in a technique called Side Eye to extract audio from static photos and silent videos.

This technique targets the image stabilization technology that is standard across most modern smartphones. Cameras have small springs that hold the lens suspended in liquid, and an electromagnet and sensors then push the lens in equal and opposite directions to reduce camera shake and prevent blurry photos.

According to Cybernews, the “Side Eye” technique analyzes how camera lenses adjust to movements caused by sound waves and extracts that information from the recorded photo or video to reproduce the original sound. Kevin Fu, professor of electrical and computer engineering and computer science at Northeastern University, explains that the audio reconstruction accuracy varies from 80% to 99% based on the amount and complexity of the sound that’s reconstructed.

Researchers clarify that Side Eye does not currently work with human speech and was only tested with sound from powerful speakers. Nevertheless, it is easy to imagine the possibilities in the future, and it may be quite dangerous from a cybersecurity perspective.

The researchers explain: “Our analysis and experiments with ten smartphones demonstrate how malicious parties with knowledge of camera hardware structure can extract fine-grained acoustic information from recorded videos, achieving digit, speaker, and gender recognition.”

Despite the widespread belief that smartphone users are protected from stolen audio when microphone access is disabled, this technology proves that granting camera access to apps opens the possibility of acoustic eavesdropping. The resulting audio is currently very muffled, but researchers claim that machine learning and AI can help clear it up and turn it into usable information.

When it comes to countermeasures, the paper recommends using lower-quality cameras and holding the devices away from speakers or adding vibration-isolation-dampening materials. They also state that manufacturers keen on securing future camera devices should focus on camera design improvement.