AI Able to Generate Video from Brain Activity

This post is also available in: עברית (Hebrew)

How many times have you wished you could play back your dream on your computer or phone? With this new discovery, the technology might be closer than you think.

In a research published last week on the arXiv server, researchers at the National University of Singapore and the Chinese University of Hong Kong reported that they have developed a process capable of generating video from brain scans.

According to Techxplore, using functional magnetic resonance imaging (fMRI), researchers Jiaxin Qing, Zijiao Chen and Juan Helen Zhou coupled data retrieved through imaging with the deep learning model Stable Diffusion to create smooth, high-quality videos.

“Understanding the information hidden within our complex brain activities is a big puzzle in cognitive neuroscience,” Qing said. “The task of recreating human vision from brain recordings, especially using non-invasive tools like fMRI, is an exciting but difficult task.”

His team achieved high-caliber videos with their Mind-Video model. Described as “a two-module pipeline designed to bridge the gap between image and video brain decoding,” its fMRI decoder progressively learns from acquired brain signals, training with image databases and fine-tuning. The results were high-quality videos complete with motion and scene dynamics at an accuracy rate of 85%.

The use of AI along with MRI and EGM (electromyogram) to study imagery, brain activity and muscle movement is opening new horizons into the workings of the mind. These processes may one day be used to capture thoughts and dreams.

Dream researcher Daniel Oldis, working at the Cognitive Neuroscience Lab at the University of Texas, is working on dream-recording MRI technology that tracks brain activity and nerve impulses in muscles to define images, speech and movement in dreams.

“This is like the early years of the space race,” he said recently. “But in this case, we’re going into the dream space.”