Meta decodes images from brain activity in real-time

From Superinnovators 25/10/23. This article is in TLDR (Too Long Didn’t Read) format which is popular in the innovation community and provides a bullet summary of information.

Meta’s milestone: Meta has developed an AI system using magnetoencephalography (MEG) that can decode the visual representations in the brain with exceptional temporal precision.

AI system capabilities: This system can reconstruct images perceived by the brain in real-time, offering insights into how images form the basis of human intelligence. This technology might eventually aid in non-invasive brain-computer interfaces, especially for individuals who’ve lost speech due to brain lesions.

Three-part system: Meta’s system comprises an image encoder, a brain encoder, and an image decoder. The image encoder interprets the image separately from the brain, the brain encoder aligns MEG signals to these image embeddings, and the image decoder reconstructs a feasible image from these brain representations.

Training and comparison: The system is trained on a public dataset of MEG recordings from healthy volunteers. When comparing its performance with other image modules, it was found that brain signals align best with AI models like DINOv2, which learns visual representations without human annotations. This suggests that self-supervised AI systems align closely with brain-like representations.

Imperfections in generated images: While the AI system can generate images that preserve high-level features, there are inaccuracies in low-level features, often misplacing or mis-orienting objects. Images derived from MEG decoding are less precise than those obtained with fMRI, which provides slower but more spatially accurate imaging.

Complexity decoded: Meta’s research demonstrates that MEG and AI can be used to decode the complex representations generated in the brain with millisecond precision. This work further solidifies Meta’s commitment to understanding human intelligence and developing AI systems that think and reason similarly to humans.

MEG recordings are continuously aligned to the deep representation of the images, which can then condition the generation of images at each instant. Credit: Meta

Image decoded from MEG in real-time. Credit: Meta

Image decoded from fMRI after imaged viewed by patient i.e. not in real-time. Credit: Meta

Example of MEG scanner with patient from National Institute of Mental Health. Image not taken as part of Meta project. Credit: National Institute of Mental Health, National Institutes of Health, Department of Health and Human Services

Example of functional Magnetic Resonance Imaging (fMRI) machine at the Imperial Centre for Psychedelic Research. Image not taken as part of Meta project. Credit: Thomas Angus, Imperial College London