Cookies Policy
X

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Full Access Where’s Wally? Audio–visual mismatch directs ocular saccades in sensory substitution

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Where’s Wally? Audio–visual mismatch directs ocular saccades in sensory substitution

  • HTML
  • PDF
Add to Favorites
You must be logged in to use this functionality

image of Seeing and Perceiving
For more content, see Multisensory Research and Spatial Vision.

Sensory substitution is the representation of information from one sensory modality (e.g., vision) within another modality (e.g., audition). We used a visual-to-auditory sensory substitution device (SSD) to explore the effect of incongruous (true-)visual and substituted-visual signals on visual attention. In our multisensory sensory substitution paradigm, both visual and sonified-visual information were presented. By making small alterations to the sonified image, but not the seen image, we introduced audio–visual mismatch. The alterations consisted of the addition of a small image (for instance, the Wally character from the ‘Where’s Wally?’ books) within the original image. Participants were asked to listen to the sonified image and identify which quadrant contained the alteration. Monitoring eye movements revealed the effect of the audio–visual mismatch on covert visual attention. We found that participants consistently fixated more, and dwelled for longer, in the quadrant corresponding to the location (in the sonified image) of the target. This effect was not contingent on the participant reporting the location of the target correctly, which indicates a low-level interaction between an auditory stream and visual attention. We propose that this suggests a shared visual workspace that is accessible by visual sources other than the eyes. If this is indeed the case, it would support the development of other, more esoteric, forms of sensory substitution. These could include an expanded field of view (e.g., rear-view cameras), overlaid visual information (e.g., thermal imaging) or restoration of partial visual field loss (e.g., hemianopsia).

Affiliations: 1: University of Sussex, GB

Sensory substitution is the representation of information from one sensory modality (e.g., vision) within another modality (e.g., audition). We used a visual-to-auditory sensory substitution device (SSD) to explore the effect of incongruous (true-)visual and substituted-visual signals on visual attention. In our multisensory sensory substitution paradigm, both visual and sonified-visual information were presented. By making small alterations to the sonified image, but not the seen image, we introduced audio–visual mismatch. The alterations consisted of the addition of a small image (for instance, the Wally character from the ‘Where’s Wally?’ books) within the original image. Participants were asked to listen to the sonified image and identify which quadrant contained the alteration. Monitoring eye movements revealed the effect of the audio–visual mismatch on covert visual attention. We found that participants consistently fixated more, and dwelled for longer, in the quadrant corresponding to the location (in the sonified image) of the target. This effect was not contingent on the participant reporting the location of the target correctly, which indicates a low-level interaction between an auditory stream and visual attention. We propose that this suggests a shared visual workspace that is accessible by visual sources other than the eyes. If this is indeed the case, it would support the development of other, more esoteric, forms of sensory substitution. These could include an expanded field of view (e.g., rear-view cameras), overlaid visual information (e.g., thermal imaging) or restoration of partial visual field loss (e.g., hemianopsia).

Loading

Full text loading...

/deliver/18784763/25/0/18784763_025_00_S057_text.html;jsessionid=04mmE8JnexIit4x76zkVwThy.x-brill-live-03?itemId=/content/journals/10.1163/187847612x646820&mimeType=html&fmt=ahah
/content/journals/10.1163/187847612x646820
Loading

Data & Media loading...

http://brill.metastore.ingenta.com/content/journals/10.1163/187847612x646820
Loading
Loading

Article metrics loading...

/content/journals/10.1163/187847612x646820
2012-01-01
2016-12-06

Sign-in

Can't access your account?
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation