Our brains are amazing organs. They are capable of receiving, synthesizing, and integrating massive amounts of information in a way that lets us effectively live our day to day lives. Our brains constantly integrate sense data from all of our senses to give us a seamless experience of the world around us.
Unfortunately, our brain’s sensory capabilities can be hampered through medical conditions and this sense disruption can significantly change how we experience the world. Hearing loss and sight loss are two significant ways in which our lived experience of the world can change dramatically.
While most people are concerned about how hearing loss or sight loss will affect their ability to hear and see, respectively, few people know about the effect that hearing loss can have on your sight, and vice versa. A new study from researchers Christopher C. Berger and H. Henrik Ehrsson at the Swedish Karolinska Institut and the California Institute of Technology aims to better understand this ‘cross-modal’ sensory effect.
The study
Consider this – you’re sitting in a cinema before the movie starts, anxiously awaiting the upcoming feature film. The lights go dark and the movie is starting. No images appear on the screen just yet, but a voice speaks over the stereo system. Your ears are quickly able to locate the sound of the voice – it’s coming out of the speakers on the walls behind you.
Then, the screen lights up and the movie begins. Immediately, your body recognizes that the sounds are now coming from the actors on the screen, despite the fact that the screen itself makes no noise – the speakers do.
Although you might never have noticed this before, this perceived sound location shift is an effect known as the ventriloquist illusion. Essentially, when visual and audible stimuli are presented at the same time, we perceive that the sound is coming from the location of the visual stimulus, even if it isn’t. If this pairing is repeated a number of times, this illusion will continue even when the visual stimulus disappears and the only thing we experience is the sound.
While we generally believe that our hearing and listening abilities are separate and we often think that we can differentiate between stimuli from the real and imagined worlds, brain imaging data from a number of studies indicates that the distinction between the audible and visual and the distinction between the real and the imagined is not so clear.
During their study, Berger and Ehrsson asked participants to imagine a circle on a computer screen while listening to a series of noises which came from one of three locations behind the screen (right, left, or center). The participants were tasked with identifying where the noise came from – either right, left, or center. The results of the tests show that imagining a visual stimulus while hearing a sound was enough to create this ventriloquism illusion. This illusion, researchers note, was nearly as strong for an imagined visual stimulus as it is for a real one.
Although this might seem like something out of a Sci-Fi novel, the potential implications for this type of research are vast. The research is still in its early stages, however, studies on the cross-modal connections between visual and sensory inputs can have significant effects on the medical community.
The researchers suggest that their work could eventually be used to help in the rehabilitation of stroke patients or the development of neural prostheses. Plus, more information on the connection between hearing and sight could help medical professionals in the treatment of people with hearing or sight loss. The potential impacts of this work on how our different senses are linked could reach far and wide, so it will be exciting to see how this research develops in the future.
