Brain implants to restore sight, like Neuralink’s Blindsight, face a fundamental problem − more pixels don’t ensure better vision
- Written by Ione Fine, Professor of Psychology, University of Washington
Elon Musk recently pronounced[1] that the next Neuralink project will be a “Blindsight” cortical implant to restore vision: “Resolution will be low at first, like early Nintendo graphics, but ultimately may exceed normal human vision.”
Unfortunately, this claim rests on the fallacy that neurons in the brain are like pixels on a screen[2]. It’s not surprising that engineers often assume that “more pixels equals better vision.” After all, that is how monitors and phone screens work.
In our newly published research, we created a computational model of human vision[3] to simulate what sort of vision an extremely high-resolution cortical implant might provide. A movie of a cat with a resolution of 45,000 pixels is sharp and clear. A movie generated using a simplified version of a model of 45,000 cortical electrodes, each of which stimulates a single neuron, still has a recognizable cat but most of the details of the scene are lost.
The movie on the left is generated using 45,000 pixels. The one on the right is generated using a simulation of a cortical prosthesis with 45,000 electrodes, each of which stimulates a single neuron.The reason why the movie generated by electrodes is so blurry is because neurons in the human visual cortex do not represent tiny dots or pixels. Instead, each neuron has a particular receptive field[4], which is the location and pattern a visual stimulus must have in order to make that neuron fire. Electrically stimulating a single neuron produces a blob whose appearance is determined by that neuron’s receptive field. The tiniest electrode – one that stimulates a single neuron – will produce a blob that is roughly the size of your pinkie’s width held at arm’s length.
Consider what happens when you look at a single star in the night sky. Each point in space is represented by many thousands of neurons with overlapping receptive fields. A tiny spot of light, such as a star, results in a complex pattern of firing across all these neurons.
A star in the night sky elicits spikes in hundreds of neurons, each of which has a different receptive field.
Ione Fine and Geoffrey Boynton/University of Washington
To generate the visual experience of seeing a single star with cortical stimulation, you would need to reproduce a pattern of neural responses that is similar to the pattern that would be produced by natural vision.
Some scientists have suggested that generating the correct stimulation pattern could make it possible to restore naturalistic vision. But it is still unclear how this could be achieved.
Ione Fine and Geoffrey Boynton/University of Washington
In order to do this, you would obviously need thousands of electrodes. But you would also need to replicate the correct pattern of neuronal responses, which requires knowing every neuron’s receptive field. Our simulations show that knowing the location of each neuron’s receptive field in space is not enough – if you don’t also know the orientation and size of each receptive field, then the star becomes a fuzzy mess.
If the orientation and size of the receptive field for each cell is unknown, the resulting image no longer looks like a star, despite using the same electrode stimulation protocol.
Ione Fine and Geoffrey Boynton/University of Washington
So, even a single star – a single, bright pixel – generates an immensely complex neural response in the visual cortex. Imagine the even more complex pattern of cortical stimulation needed to accurately reproduce natural vision.
Some scientists have suggested that by stimulating exactly the right combination of electrodes[5], it would be possible to produce natural vision[6]. Unfortunately, no one has yet suggested a sensible way to determine the receptive field of each individual neuron in a specific blind patient. Without that information, there is no way to see the stars. Vision from cortical implants will remain grainy and imperfect, regardless of the number of electrodes.
Sight restoration is not simply an engineering problem. Predicting what kind of vision a device will provide requires knowing how the technology interfaces with the complexities of the human brain[7].
How we created our virtual patients
In our work as computational[8] neuroscientists[9], we develop simulations that predict the perceptual experience of patients seeking to restore their sight.
We have previously created a model to predict the perceptual experience of patients with a retinal implant[10]. To create a virtual patient to predict what cortical implant patients would see, we simulated the neurophysiological architecture of the area of the brain involved in the first stage of visual processing[11]. Our model approximates how receptive fields increase in size from central to peripheral vision and the fact that every neuron has a unique receptive field.
Our model successfully predicted[12] data describing the perceptual experience of participants across a wide range of studies on cortical stimulation in people. Having confirmed that our model could predict existing data, we used it to make predictions about the quality of vision that possible future cortical implants might produce.
Models like ours are an example of virtual prototyping[13], which involves using computer systems to improve product design. These models can facilitate new technology development and evaluate device performance. Our study shows they can also offer more realistic expectations about what kind of vision bionic eyes might provide.
First do no harm
In our almost 20 years researching bionic eyes, we’ve seen the complexity of the human brain defeat company after company. Patients pay the cost[14] when these devices fail, left stranded with orphaned technologies in their eye or brain.
The Food and Drug Administration could mandate that sight recovery tech companies must develop failure plans that minimize harm[15] to patients when technologies stop working. Possibilities include requiring companies implanting neuroelectronic devices into patients to participate in technology escrow agreements[16] and carry insurance to ensure continuing medical care[17] and technology support[18] if they go bankrupt.
If cortical implants can achieve anything close to the resolution of our simulations, that would still be an accomplishment worth celebrating. Grainy and imperfect vision would be life-changing for many thousands of people who currently suffer from incurable blindness. But this is a moment for cautious rather than blind optimism.
References
- ^ recently pronounced (x.com)
- ^ fallacy that neurons in the brain are like pixels on a screen (doi.org)
- ^ computational model of human vision (doi.org)
- ^ receptive field (dx.doi.org)
- ^ right combination of electrodes (doi.org)
- ^ produce natural vision (doi.org)
- ^ technology interfaces with the complexities of the human brain (doi.org)
- ^ computational (scholar.google.com)
- ^ neuroscientists (scholar.google.com)
- ^ perceptual experience of patients with a retinal implant (doi.org)
- ^ first stage of visual processing (dx.doi.org)
- ^ Our model successfully predicted (doi.org)
- ^ virtual prototyping (doi.org)
- ^ Patients pay the cost (spectrum.ieee.org)
- ^ failure plans that minimize harm (spectrum.ieee.org)
- ^ technology escrow agreements (www.nextgov.com)
- ^ medical care (spectrum.ieee.org)
- ^ technology support (spectrum.ieee.org)
Authors: Ione Fine, Professor of Psychology, University of Washington
