Cookies Policy
X

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Open Access Visual Objects in the Auditory System in Sensory Substitution: How Much Information Do We Need?

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Visual Objects in the Auditory System in Sensory Substitution: How Much Information Do We Need?

  • HTML
  • PDF
Add to Favorites

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Sensory substitution devices such as The vOICe convert visual imagery into auditory soundscapes and can provide a basic ‘visual’ percept to those with visual impairment. However, it is not known whether technical or perceptual limits dominate the practical efficacy of such systems. By manipulating the resolution of sonified images and asking naïve sighted participants to identify visual objects through a six-alternative forced-choice procedure (6AFC) we demonstrate a ‘ceiling effect’ at 8 × 8 pixels, in both visual and tactile conditions, that is well below the theoretical limits of the technology. We discuss our results in the context of auditory neural limits on the representation of ‘auditory’ objects in a cortical hierarchy and how perceptual training may be used to circumvent these limitations.

Affiliations: 1: 3Centre for Digital Music, Queen Mary University of London, UK; 2: 1Crossmodal Cognition Lab, Department of Psychology, 2 South, University of Bath, Bath, BA2 7AY, UK

Sensory substitution devices such as The vOICe convert visual imagery into auditory soundscapes and can provide a basic ‘visual’ percept to those with visual impairment. However, it is not known whether technical or perceptual limits dominate the practical efficacy of such systems. By manipulating the resolution of sonified images and asking naïve sighted participants to identify visual objects through a six-alternative forced-choice procedure (6AFC) we demonstrate a ‘ceiling effect’ at 8 × 8 pixels, in both visual and tactile conditions, that is well below the theoretical limits of the technology. We discuss our results in the context of auditory neural limits on the representation of ‘auditory’ objects in a cortical hierarchy and how perceptual training may be used to circumvent these limitations.

Loading

Full text loading...

/deliver/journals/22134808/27/5-6/22134808_027_05-06_s005_text.html;jsessionid=3f2hhq9r5qhkf.x-brill-live-02?itemId=/content/journals/10.1163/22134808-00002462&mimeType=html&fmt=ahah
/content/journals/10.1163/22134808-00002462
Loading

Data & Media loading...

1. Abboud S. , Hanassy S. , Levy-Tzedek S. , Maidenbaum S. , Amedi A. ( 2014). "EyeMusic: introducing a “visual” colorful experience for the blind using auditory sensory substitution", Rest. Neurol. Neurosci. Vol 32, 247257.
2. Ahissar M. , Hochstein S. ( 2004). "The reverse hierarchy theory of visual perceptual learning", Trends Cogn. Sci. Vol 8, 457464. http://dx.doi.org/10.1016/j.tics.2004.08.011
3. Ahissar M. , Nahum M. , Nelken I. , Hochstein S. ( 2009). "Reverse hierarchies and sensory learning", Phil. Trans. R. Soc. B Vol 364, 285299. http://dx.doi.org/10.1098/rstb.2008.0253
4. Amedi A. , Stern W. M. , Camprodon J. A. , Bermpohl F. , Merabet L. , Rotman S. , Hemond C. , Meijer P. , Pascual-Leone A. ( 2007). "Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex", Nat. Neurosci. Vol 10, 687689. http://dx.doi.org/10.1038/nn1912
5. Arno P. , De Volder A. G. , Vanlierde A. , Wanet-Defalque M. C. , Streel E. , Robert A. , Sanabria-Bohórquez S. , Veraart C. ( 2001). "Occipital activation by pattern recognition in the early blind using auditory substitution for vision", Neuroimage Vol 13, 632645. http://dx.doi.org/10.1006/nimg.2000.0731
6. Auvray M. , Hanneton S. , O’Regan J. K. ( 2007). "Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with ‘The vOICe’", Perception Vol 36, 416430. http://dx.doi.org/10.1068/p5631
7. Bach-y-Rita P. ( 2004). "Tactile sensory substitution studies", Coevol. Hum. Potential Converging Technol. Vol 1013, 8391.
8. Bach-y-Rita P. , Kercel S. W. ( 2003). "Sensory substitution and the human–machine interface", Trends Cogn. Sci. Vol 7, 541546. http://dx.doi.org/10.1016/j.tics.2003.10.013
9. Bach-y-Rita P. , Collins C. C. , Saunders F. A. , White B. , Scadden L. ( 1969). "Vision substitution by tactile image projection", Nature Vol 221( 5184), 963964. http://dx.doi.org/10.1038/221963a0
10. Barton B. , Venezia J. H. , Saberi K. , Hickok G. , Brewer A. A. ( 2012). "Orthogonal acoustic dimensions define auditory field maps in human cortex", Proc. Natl Acad. Sci. USA Vol 109, 2073820743. http://dx.doi.org/10.1073/pnas.1213381109
11. Baumann S. , Griffiths T. D. , Sun L. , Petkov C. I. , Thiele A. , Rees A. ( 2011). "Orthogonal representation of sound dimensions in the primate midbrain", Nat. Neurosci. Vol 14, 423425. http://dx.doi.org/10.1038/nn.2771
12. Benav H. , Bartz-Schmidt K. U. , Besch D. , Bruckmann A. , Gekeler F. , Greppmaier U. , Harscher A. , Kibbel S. , Kusnyerik A. , Peters T. , Sachs H. , Stett A. , Stingl K. , Wilhelm B. , Wilke R. , Wrobel W. , Zrenner E. ( 2010). Restoration of useful vision up to letter recognition capabilities using subretinal microphotodiodes, Conf. Proc. IEEE Eng. Med. Biol. Soc. 2010, 5919–5922.
13. Brindley G. S. , Lewin W. S. ( 1968a). "The sensations produced by electrical stimulation of the visual cortex", J. Physiol. Vol 196, 479493. http://dx.doi.org/10.1113/jphysiol.1968.sp008519
14. Brindley G. S. , Lewin W. S. ( 1968b). "The visual sensations produced by electrical stimulation of the medial occipital cortex", J. Physiol. Vol 194, 5455P.
15. Brown D. J. , Macpherson T. , Ward J. ( 2011). "Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device", Perception Vol 40, 11201135. http://dx.doi.org/10.1068/p6952
16. Brown D. J. , Proulx M. J. ( 2013). "Increased signal complexity improves the breadth of generalization in auditory perceptual learning", Neural Plast. 879047. DOI: .
17. Capelle C. , Trullemans C. , Arno P. , Veraart C. ( 1998). "A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution", IEEE Trans. Biomed. Eng. Vol 45, 12791293. http://dx.doi.org/10.1109/10.720206
18. Chai X. , Li L. , Wu K. , Zhou C. , Cao P. , Ren Q. ( 2008a). "C-sight visual prostheses for the blind", IEEE Eng. Med. Biol. Mag. Vol 27( 5), 2028. http://dx.doi.org/10.1109/MEMB.2008.923959
19. Chai X. , Zhang L. , Li W. , Shao F. , Yang K. , Ren Q. ( 2008b). "Study of tactile perception based on phosphene positioning using simulated prosthetic vision", Artif Organs Vol 32, 110115. http://dx.doi.org/10.1111/j.1525-1594.2007.00469.x
20. Chebat D. R. , Rainville C. , Kupers R. , Ptito M. ( 2007). "Tactile-‘visual’ acuity of the tongue in early blind individuals", Neuroreport Vol 18, 19011904. http://dx.doi.org/10.1097/WNR.0b013e3282f2a63
21. Collignon O. , Lassonde M. , Lepore F. , Bastien D. , Veraart C. ( 2007). "Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects", Cereb. Cortex Vol 17, 457465. http://dx.doi.org/10.1093/cercor/bhj162
22. Craddock M. , Lawson R. ( 2008). "Repetition priming and the haptic recognition of familiar and unfamiliar objects", Percept. Psychophys. Vol 70, 13501365. http://dx.doi.org/10.3758/PP.70.7.1350
23. Danilov Y. P. , Tyler M. ( 2005). "Brainport: an alternative input to the brain", J. Integr. Neurosci. Vol 4, 537550. http://dx.doi.org/10.1142/S0219635205000914
24. Danilov Y. P. , Tyler M. E. , Skinner K. L. , Hogle R. A. , Bach-y-Rita P. ( 2007). "Efficacy of electrotactile vestibular substitution in patients with peripheral and central vestibular loss", J. Vestib. Res. Vol 17, 119130.
25. Ding N. , Simon J. Z. ( 2012). "Emergence of neural encoding of auditory objects while listening to competing speakers", Proc. Natl Acad. Sci. USA Vol 109, 1185411859. http://dx.doi.org/10.1073/pnas.1205381109
26. Dobelle W. H. , Mladejovsky M. G. ( 1974). "Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind", J. Physiol. Vol 243, 553576. http://dx.doi.org/10.1113/jphysiol.1974.sp010766
27. Dobelle W. H. , Mladejovsky M. G. , Girvin J. P. ( 1974). "Artifical vision for the blind: electrical stimulation of visual cortex offers hope for a functional prosthesis", Science Vol 183( 4123), 440444. http://dx.doi.org/10.1126/science.183.4123.440
28. Eickenscheidt M. , Jenkner M. , Thewes R. , Fromherz P. , Zeck G. ( 2012). "Electrical stimulation of retinal neurons in epiretinal and subretinal configuration using a multicapacitor array", J. Neurophysiol. Vol 107, 27422755. http://dx.doi.org/10.1152/jn.00909.2011
29. Fujikado T. , Kamei M. , Sakaguchi H. , Kanda H. , Morimoto T. , Ikuno Y. , Nishida K. , Kishima H. , Maruo T. , Konoma K. , Ozawa M. , Nishida K. ( 2011). "Testing of semichronically implanted retinal prosthesis by suprachoroidal-transretinal stimulation in patients with retinitis pigmentosa", Invest. Ophthalmol. Vis. Sci. Vol 52, 47264733. http://dx.doi.org/10.1167/iovs.10-6836
30. Haigh A. , Brown D. J. , Meijer P. , Proulx M. J. ( 2013). "How well do you see what you hear? The acuity of visual-to-auditory sensory substitution", Front. Psychol. Vol 4, 330. http://dx.doi.org/10.3389/fpsyg.2013.00330
31. Humphries C. , Liebenthal E. , Binder J. R. ( 2010). "Tonotopic organization of human auditory cortex", Neuroimage Vol 50, 12021211. http://dx.doi.org/10.1016/j.neuroimage.2010.01.046
32. Joris P. X. , Schreiner C. E. , Rees A. ( 2004). "Neural processing of amplitude-modulated sounds", Physiol. Rev. Vol 84, 541577. http://dx.doi.org/10.1152/physrev.00029.2003
33. Keseru M. , Feucht M. , Bornfeld N. , Laube T. , Walter P. , Rossler G. , Velikay-Parel M. , Hornig R. , Richard G. ( 2012). "Acute electrical stimulation of the human retina with an epiretinal electrode array", Acta Ophthalmol. Vol 90, e1e8. DOI: . http://dx.doi.org/10.1111/j.1755-3768.2011.02288.x
34. Li S. , Hu J. , Chai X. Y. , Peng Y. H. ( 2012). "Image recognition with a limited number of pixels for visual prostheses design", Artif. Organs Vol 36, 266274. http://dx.doi.org/10.1111/j.1525-1594.2011.01347.x
35. Martinovic J. , Gruber T. , Hantsch A. , Muller M. M. ( 2008). "Induced gamma-band activity is related to the time point of object identification", Brain Res. Vol 1198, 93106. http://dx.doi.org/10.1016/j.brainres.2007.12.050
36. Meijer P. B. L. ( 1992). "An experimental system for auditory image representations", IEEE Trans. Biomed. Eng. Vol 39, 112121. http://dx.doi.org/10.1109/10.121642
37. Mesgarani N. , Chang E. F. ( 2012). "Selective cortical representation of attended speaker in multi-talker speech perception", Nature Vol 485( 7397), 233236. http://dx.doi.org/10.1038/nature11020
38. Normann R. A. , Maynard E. M. , Rousche P. J. , Warren D. J. ( 1999). "A neural interface for a cortical vision prosthesis", Vis. Res. Vol 39, 25772587. http://dx.doi.org/10.1016/S0042-6989(99)00040-1
39. Overvliet K. E. , Smeets J. B. , Brenner E. ( 2007). "Parallel and serial search in haptics", Percept. Psychophys. Vol 69, 10591069. http://dx.doi.org/10.3758/BF03193944
40. Pascolini D. , Mariotti S. P. ( 2012). "Global estimates of visual impairment: 2010", Br. J. Ophthalmol. Vol 96, 614618. http://dx.doi.org/10.1136/bjophthalmol-2011-300539
41. Pasley B. N. , David S. V. , Mesgarani N. , Flinker A. , Shamma S. A. , Crone N. E. , Knight R. T. , Chang E. F. ( 2012). "Reconstructing speech from human auditory cortex", PLoS Biol. Vol 10( 1), e1001251. DOI: . http://dx.doi.org/10.1371/journal.pbio.1001251
42. Pasqualotto A. , Proulx M. J. ( 2012). "The role of visual experience for the neural basis of spatial cognition", Neurosci. Biobehav. Rev. Vol 36, 11791187. http://dx.doi.org/10.1016/j.neubiorev.2012.01.008
43. Pasqualotto A. , Lam J. S. , Proulx M. J. ( 2013a). "Congenital blindness improves semantic and episodic memory", Behav. Brain Res. Vol 244, 162165. http://dx.doi.org/10.1016/j.bbr.2013.02.005
44. Pasqualotto A. , Spiller M. J. , Jansari A. S. , Proulx M. J. ( 2013b). "Visual experience facilitates allocentric spatial representation", Behav. Brain Res. Vol 236, 175179. http://dx.doi.org/10.1016/j.bbr.2012.08.042
45. Pasqualotto A. , Taya S. , Proulx M. J. ( 2014). "Sensory deprivation: visual experience alters the mental number line", Behav. Brain Res. Vol 261, 110113. http://dx.doi.org/10.1016/j.bbr.2013.12.017
46. Plaza P. , Cuevas I. , Grandin C. , De Volder A. G. , Renier L. ( 2012). "Looking into task-specific activation using a prosthesis substituting vision with audition", ISRN Rehabil. Vol 2012, 15. DOI: . http://dx.doi.org/10.5402/2012/490950
47. Poirier C. C. , De Volder A. G. , Tranduy D. , Scheiber C. ( 2006). "Neural changes in the ventral and dorsal visual streams during pattern recognition learning", Neurobiol. Learn. Mem. Vol 85, 3643. http://dx.doi.org/10.1016/j.nlm.2005.08.006
48. Proulx M. J. , Stoerig P. , Ludowig E. , Knoll I. ( 2008). "Seeing ‘where’ through the ears: effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution", Plos One Vol 3( 3), e1840. DOI: . http://dx.doi.org/10.1371/journal.pone.0001840
49. Proulx M. J. , Brown D. J. , Pasqualotto A. , Meijer P. ( 2014). "Multisensory perceptual learning and sensory substitution", Neurosci. Biobehav. Rev. Vol 41, 1625. http://dx.doi.org/10.1016/j.neubiorev.2012.11.017
50. Reuschel J. , Rosler F. , Henriques D. Y. P. , Fiehler K. ( 2012). "Spatial updating depends on gaze direction even after loss of vision", J. Neurosci. Vol 32, 24222429. http://dx.doi.org/10.1523/JNEUROSCI.2714-11.2012
51. Sampaio E. , Maris S. , Bach-y-Rita P. ( 2001). "Brain plasticity: ‘visual’ acuity of blind persons via the tongue", Brain Res. Vol 908, 204207. http://dx.doi.org/10.1016/S0006-8993(01)02667-1
52. Schmidt E. M. , Bak M. J. , Hambrecht F. T. , Kufta C. V. , O’Rourke D. K. , Vallabhanath P. ( 1996). "Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex", Brain Vol 119( Pt 2), 507522. http://dx.doi.org/10.1093/brain/119.2.507
53. Shamma S. A. , Elhilali M. , Micheyl C. ( 2011). "Temporal coherence and attention in auditory scene analysis", Trends Neurosci. Vol 34, 114123. http://dx.doi.org/10.1016/j.tins.2010.11.002
54. Simon J. Z. , Ding N. ( 2010). Magnetoencephalography and auditory neural representations, 26th Southern Biomedical Engineering Conference: Sbec 2010, 32, 45–48.
55. Simpson A. J. , Reiss J. D. ( 2013). "The dynamic range paradox: a central auditory model of intensity change detection", PLoS One Vol 8( 2), e57497. DOI: . http://dx.doi.org/10.1371/journal.pone.0057497
56. Simpson A. J. , Reiss J. D. , McAlpine D. ( 2013). "Tuning of human modulation filters is carrier-frequency dependent", PLoS One Vol 8( 8), e73590. DOI: . http://dx.doi.org/10.1371/journal.pone.0073590
57. Striem-Amit E. , Guendelman M. , Amedi A. ( 2012). "‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution", Plos One Vol 7( 3), e33136. DOI: . http://dx.doi.org/10.1371/journal.pone.0033136
58. Teki S. , Chait M. , Kumar S. , Shamma S. , Griffiths T. D. ( 2013). "Segregation of complex acoustic scenes based on temporal coherence", Elife Vol 2, e00699. DOI: . http://dx.doi.org/10.7554/eLife.00699
59. Veraart C. , Wanet-Defalque M.-C. , Gérard B. , Vanlierde A. , Delbeke J. ( 2003). "Pattern recognition with the optic nerve visual prosthesis", Artif. Organs Vol 27, 9961004. http://dx.doi.org/10.1046/j.1525-1594.2003.07305.x
60. Wallace M. T. , Perrault T. J. , Hairston W. D. , Stein B. E. ( 2004). "Visual experience is necessary for the development of multisensory integration", J. Neurosci. Vol 24, 95809584. http://dx.doi.org/10.1523/JNEUROSCI.2535-04.2004
61. Ward J. , Meijer P. ( 2010). "Visual experiences in the blind induced by an auditory sensory substitution device", Conscious. Cogn. Vol 19, 492500. http://dx.doi.org/10.1016/j.concog.2009.10.006
62. Weiland J. D. , Liu W. , Humayun M. S. ( 2005). "Retinal prosthesis", Annu. Rev. Biomed. Eng. Vol 7, 40. http://dx.doi.org/10.1146/annurev.bioeng.7.060804.100435
63. Wright B. A. , Buonomano D. V. , Mahncke H. W. , Merzenich M. M. ( 1997). "Learning and generalization of auditory temporal-interval discrimination in humans", J. Neurosci. Vol 17, 39563963.
64. Wright B. A. , Wilson R. M. , Sabin A. T. ( 2010). "Generalization lags behind learning on an auditory perceptual task", J. Neurosci. Vol 30, 1163511639. http://dx.doi.org/10.1523/JNEUROSCI.1441-10.2010
65. Xiang J. J. , Poeppel D. , Simon J. Z. ( 2013). "Physiological evidence for auditory modulation filterbanks: cortical responses to concurrent modulations", J. Acoust. Soc. Am. Vol 133, El7El12.
66. Zrenner E. , Bartz-Schmidt K. U. , Benav H. , Besch D. , Bruckmann A. , Gabel V.-P. , Gekeler F. , Greppmaier U. , Harscher A. , Kibbel S. , Koch J. , Kusnyerik A. , Peters T. , Stingl K. , Sachs H. , Stett A. , Szurman P. , Wilhelm B. , Wilke R. ( 2011). "Subretinal electronic chips allow blind patients to read letters and combine them to words", Proc. Biol. Sci. Vol 278( 1711), 14891497. http://dx.doi.org/10.1098/rspb.2010.1747
http://brill.metastore.ingenta.com/content/journals/10.1163/22134808-00002462
Loading
Loading

Article metrics loading...

/content/journals/10.1163/22134808-00002462
2014-11-11
2016-09-27

Sign-in

Can't access your account?
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation