Brain Scans to Recreate Images People Saw

While this brain-scan-to-image A.I. technology is far from ready for public use, researchers say it could someday prove useful for understanding what’s happening inside people’s minds. Once scientists refine the concept a bit more, doctors may eventually be able to use it to help people, such as those suffering from paralysis, to communicate. It might also help neuroscientists interpret dreams or even understand how other species perceive the world around them.

The researchers from Osaka University in Japan are among the ranks of scientists using A.I. to make sense of human brain scans. Their approach to this, however, is the first to use the text-to-image generator Stable Diffusion, which came on the fast-growing A.I. scene in August 2022. Their model is also much simpler, requiring only thousands, instead of millions, of parameters, or values learned during training.

The team shared more details in a new paper, which has not been peer-reviewed, published on the preprint server bioRxiv. They also plan to present their findings at an upcoming computer vision conference, according to Science.

So, how does it all work? Typically, a user inputs a word or phrase that Stable Diffusion—or other similar technologies, such as DALL-E 2 and Midjourney—transforms into an image. This process works because the A.I. technologies have studied lots of existing images and their accompanying text captions—over time, this training allows the technology to identify patterns, which it can then recreate based on a prompt.

The researchers took this training one step further, by teaching an A.I. model to link functional magnetic resonance imaging (fMRI) data with images. More specifically, the researchers used the fMRI scans of four participants who had looked at 10,000 different images of people, landscapes and objects as part of an earlier, unrelated study. They also trained a second A.I. model to link brain activity in fMRI data with text descriptions of the pictures the study participants looked at.

Together, these two models allowed Stable Diffusion to turn fMRI data into relatively accurate imitations of images that were not part of the A.I. training set. Based on the brain scans, the first model could recreate the perspective and layout that the participant had seen, but its generated images were of cloudy and nonspecific figures. But then the second model kicked in, and it could recognize what object people were looking at by using the text descriptions from the training images. So, if it received a brain scan that resembled one from its training marked as a person viewing an airplane, it would put an airplane into the generated image, following the perspective from the first model. The technology achieved roughly 80 percent accuracy.

 The original images (left) and A.I.-generated images for all four participants. Takagi and Nishimoto / bioRxiv, 2022 under CC BY 4.0

Indeed, the recreated images look eerily similar to the originals, albeit with some noticeable differences. The A.I.-generated version of a locomotive, for example, is shrouded in a murky gray fog, rather than showing the cheery, bright blue skies of the actual image. And the A.I.’s depiction of a clock tower looks more like an abstract work of art than an actual photograph of one.