Paul Bogdan
The human brain goes through a series of steps when it processes visual stimuli into images. We see, and we store what we’ve just seen, but what happens in between is a complex process.
What if we could produce real-life images of what our brain has seen? Can you take a photo of a memory?
Scientists at the USC Viterbi Ming Hsieh Department of Electrical and Computer Engineering are trying to replicate that human visual process – brain to imagery – in the lab. A team led by Paul Bogdan, the Jack Munushian Early Career Chair and associate professor of Electrical and Computer Engineering, has developed a novel neural network architecture designed to build images from the data captured by fMRI (functional magnetic resonance imaging) scans. The scans are used to measure brain activity based on changes in blood flow in the brain.
The paper, titled “Generative Decoding of Visual Stimuli,” was presented on Wednesday by PhD student Panagiotis Kyriakis at the 40th International Conference on Machine Learning (ICML) in Honolulu. In addition to Bogdan and Kyriakis, the authors were Andrei Irimia of USC; Eleni Miliotou; and Jason Hinman of UCLA.
Panagiotis Kyriakis
“Reconstructing natural images from fMRI recordings is a challenging task of great importance in neuroscience,” the authors wrote. “The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain.
“By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able to construct a latent variable neural decoding model that replicates the hierarchical visual information processing.”
Bogdan and Kyriakis said the tools being developed could be used to decode dreams and help people suffering from depression. They could also be used in law enforcement.
“Imagine you have somebody who is a witness of a crime, but it is really painful for that person to recall the experience and try to remember things and then tell them to investigators,” Bogdan said. “So what you could do is put a person into a kind of sedative state and try to recreate what they saw with images.”
The paper was funded by the National Science Foundation and the Defense Advanced Research Projects Agency (DARPA).
Published on July 27th, 2023
Last updated on August 9th, 2023