Cookies Policy
X

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.

Affiliations: 1: School of Psychology and Sackler Centre for Consciousness Science, University of Sussex, Brighton, UK

Loading data from figshare Loading data from figshare
Loading

Full text loading...

/content/journals/10.1163/22134808-00002519
Loading

Data & Media loading...

1. Abboud S. , Hanassy S. , Levy-Tzedek S. , Maidenbaum S. , Amedi A. (2014). "EyeMusic: introducing a ‘visual’ colorful experience for the blind using auditory sensory substitution", Restor. Neurol. Neurosci. Vol 32, 247257.
2. Amedi A. , Chebat D. R. , Levy-Tzedek S. , Buchs G. , Maidenbaum S. (2014). "Returning sensory substitution to practical visual rehabilitation", Invest. Ophthalmol. Vis. Sci. Vol 55, 4146.
3. Ancuti C. , Ancuti C. , Bekaert P. (2009). ColEnViSon: color enhanced visual sonifier a polyphonic audio texture and salient scene analysis, in: VISAPP 2009: Proceedings of the 4th International Conference on Computer Vision Theory and Applications, Lisbon, Portugal, Vol. 2, pp. 566–572.
4. Bien N. , ten Oever S. , Goebel R. , Sack A. T. (2012). "The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study", Neuroimage Vol 59, 663672. http://dx.doi.org/10.1016/j.neuroimage.2011.06.095
5. Blauert J. (1997). Spatial Hearing. The Psychophysics of Human Sound Localization. MIT Press, Cambridge, MA, USA.
6. Bologna G. , Deville B. , Pun T. , Vinckenbosch A. (2007). "Transforming 3D coloured pixels into musical instrument notes for vision substitution applications", EURASIP J. Image Video Proc. Vol 2007, 076204. DOI:.
7. Bologna G. , Deville B. , Pun T. (2010). Blind navigation along a sinuous path by means of the See ColOr interface, in: Proceedings of the 3rd International Work-Conference on the Interplay Between Natural and Artifical Computation: Part II: Bioinspired Applications in Artifical and Natural Computation, Galicia, Spain, pp. 235–243.
8. Brewer C. A. (1999). Color use guidelines for data representation, in: Proceedings of the Section on Statistical Graphics, American Statistical Association, pp. 55–60.
9. Brown D. , Macpherson T. , Ward J. (2011). "Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device", Perception Vol 40, 11201135. http://dx.doi.org/10.1068/p6952
10. Brown D. J. , Simpson A. J. , Proulx M. J. (2014). "Visual objects in the auditory system in sensory substitution: how much information do we need?", Multisens. Res. Vol 27, 337357. http://dx.doi.org/10.1163/22134808-00002462
11. Burch D. (2012). Development of a multiple contact haptic display with texture-enhanced graphics, PhD Thesis, Virginia Commonwealth University, Richmond, VA, USA. Retrieved from https://digarchive.library.vcu.edu/handle/10156/3836.
12. Bush M. (1999). Alternative marking schemes for on-line multiple choice tests, in: 7th Annual Conference on the Teaching of Computing, Belfast, UK.
13. Capalbo Z. , Glenney B. (2009). Hearing color: radical pluralistic realism and SSDs, in: Proceedings of AP-CAP, October 2009, Tokyo, Japan, pp. 135–140.
14. Capelle C. , Trullemans C. , Arno P. , Veraart C. (1998). "A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution", IEEE Trans. Biomed. Eng. Vol 45, 12791293. http://dx.doi.org/10.1109/10.720206
15. Collignon O. , Lassonde M. , Lepore F. , Bastien D. , Veraart C. (2007). "Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects", Cereb. Cortex Vol 17, 457465. http://dx.doi.org/10.1093/cercor/bhj162
16. Deroy O. , Spence C. (2013). "Are we all born synaesthetic? Examining the neonatal synaesthesia hypothesis", Neurosci. Biobehav. Rev. Vol 37, 12401253. http://dx.doi.org/10.1016/j.neubiorev.2013.04.001
17. Doel K. (2003). SoundView: sensing color images by kinesthetic audio, in: Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA.
18. Elli G. V. , Benetti S. , Collignon O. (2014). "Is there a future for sensory substitution outside academic laboratories?", Multisens. Res. Vol 27, 271291. http://dx.doi.org/10.1163/22134808-00002460
19. Else L. (2012). "Cyborg makes art using seventh sense", New Sci. Vol 215, 50. http://dx.doi.org/10.1016/S0262-4079(12)62082-X
20. Fairchild M. D. (2005). Color Appearance Models, 2nd edn. John Wiley and Sons, Chichester, UK.
21. Froese T. , McGann M. , Bigge W. , Spiers A. , Seth A. K. (2012). "The enactive torch: a new tool for the science of perception", IEEE Trans. Haptics Vol 5, 365375. http://dx.doi.org/10.1109/TOH.2011.57
22. Giannakis K. (2001). Sound mosaics: a graphical user interface for sound synthesis based on audio–visual associations, PhD Thesis, Middlesex University, London, UK. Retrieved from http://eprints.mdx.ac.uk/6634/1/Konstantinos-sound_mosaics.phd.pdf.
23. Grey J. (1977). "Multidimensional percept scaling of musical timbres", J. Acoust. Soc. Am. Vol 61, 12701277. http://dx.doi.org/10.1121/1.381428
24. Hamilton-Fletcher G. , Ward J. (2013). "Representing colour through hearing and touch in sensory substitution devices", Multisens. Res. Vol 26, 503. http://dx.doi.org/10.1163/22134808-00002434
25. Hamilton-Fletcher G. , Ward J. (2014). What does red sound like? Establishing and applying aesthetics to sensory substitution, poster session presented at Synaesthesia in Perspective: Development, Networks, and Multisensory Processing, Hamburg, Germany. DOI:.
26. Hubbard T. L. (1996). "Synesthesia-like mappings of lightness, pitch, and melodic interval", Am. J. Psychol. Vol 109, 219238. http://dx.doi.org/10.2307/1423274
27. Hunt R. W. G. , Pointer M. R. (2011). Measuring Colour, 4th edn. John Wiley and Sons, Chichester, UK. http://dx.doi.org/10.1002/9781119975595
28. Huopaniemi J. , Zacharov N. , Karjalainen M. (1999). "Objective and subjective evaluation of head-related transfer function filter design", J. Audio Eng. Soc. Vol 47, 218239.
29. ISO (2003). Acoustics — normal equal-loudness-level contours, ISO 226:2003, International Organisation for Standardisation, Geneva, Switzerland.
30. Kahol K. , French J. , Bratton L. , Panchanathan S. (2006). Learning and perceiving colors haptically, in: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, OR, USA, pp. 173–180.
31. Levy-Tzedek S. , Riemer D. , Amedi A. (2014). "Color improves ‘visual’ acuity via sound", Front. Neurosci. Vol 8, 358. DOI:. http://dx.doi.org/10.3389/fnins.2014.00358
32. Lewkowicz D. , Turkewitz G. (1980). "Cross-modal equivalence in early infancy: auditory–visual intensity matching", Dev. Psychol. Vol 6, 597607. http://dx.doi.org/10.1037/0012-1649.16.6.597
33. Ludwig V. U. , Adachi I. , Matsuzawa T. (2011). "Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans", Proc. Natl Acad. Sci. Vol 108, 2066120665. http://dx.doi.org/10.1073/pnas.1112605108
34. Maidenbaum S. , Amedi A. (2015). Non-visual virtual interaction: can sensory substitution generically increase the accessibility of graphical virtual reality to the blind?, in: IEEE VR 3rd International Workshop on Virtual and Augmented Assistive Technology (VAAT), Arles, France, pp. 15–17.
35. Maidenbaum S. , Abboud S. , Amedi A. (2014a). "Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation", Neurosci. Biobehav. Rev. Vol 41, 315. http://dx.doi.org/10.1016/j.neubiorev.2013.11.007
36. Maidenbaum S. , Hanassy S. , Abboud S. , Buchs G. , Chebat D. R. , Levy-Tzedek S. , Amedi A. (2014b). "The ‘EyeCane’, a new electronic travel aid for the blind: technology, behavior and swift learning", Restor. Neurol. Neurosci. Vol 32, 813824.
37. Marks L. E. (1974). "On associations of light and sound: the mediation of brightness, pitch, and loudness", Am. J. Psychol. Vol 87, 173188. http://dx.doi.org/10.2307/1422011
38. Marks L. E. (1975). "On colored-hearing synesthesia: cross-modal translations of sensory dimensions", Psychol. Bull. Vol 82, 303331. http://dx.doi.org/10.1037/0033-2909.82.3.303
39. Marks L. E. (1987). "On cross-modal similarity: auditory–visual interactions in speeded discrimination", J. Exp. Psychol. Hum. Percept. Perform. Vol 13, 384394. http://dx.doi.org/10.1037/0096-1523.13.3.384
40. Martino G. , Marks L. E. (1999). "Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis", Perception Vol 28, 903923. http://dx.doi.org/10.1068/p2866
41. Meers S. , Ward K. (2004). A vision system for providing 3D perception of the environment via transcutaneous electro-neural stimulation, in: Proceedings of the Information Visualisation, 8th International Conference, pp. 546–552.
42. Meers S. , Ward K. (2005). A substitute vision system for providing 3D perception and GPS navigation via electro-tactile stimulation, in: Proceedings of the International Conference on Sensing Technology, Palmerston North, New Zealand, pp. 551–556.
43. Meijer P. (1992). "An experimental system for auditory image representations", IEEE Trans. Biomed. Eng. Vol 39, 112121. http://dx.doi.org/10.1109/10.121642
44. Melara R. D. (1989). "Dimensional interaction between color and pitch", J. Exp. Psychol. Hum. Percept. Perform. Vol 15, 6979. http://dx.doi.org/10.1037/0096-1523.15.1.69
45. Melara R. D. , O’Brien T. (1987). "Interaction between synesthetically corresponding dimensions", J. Exp. Psychol. Gen. Vol 116, 323336. http://dx.doi.org/10.1037/0096-3445.116.4.323
46. Miyahara T. , Koda A. , Sekiguchi R. , Amemiya T. (2012). "A psychological experiment on the correspondence between colors and voiced vowels in non-synesthetes", Kansei Eng. Int. J. Vol 11, 2734. http://dx.doi.org/10.5057/kei.11.27
47. Moos A. , Smith R. , Miller S. R. , Simmons D. R. (2014). "Cross-modal associations in synaesthesia: vowel colours in the ear of the beholder", i-Perception Vol 5, 132142. http://dx.doi.org/10.1068/i0626
48. Orlandatou K. (2012). The role of pitch and timbre in the synaesthetic experience, in: Proceedings of the 12th International Conference on Music Perception and Cognition and the 8th Triennial Conference of the European Society for the Cognitive Sciences of Music, Thessaloniki, Greece, pp. 751–758.
49. Palmer S. E. , Schloss K. B. , Xu Z. , Prado-León L. R. (2013). "Music–color associations are mediated by emotion", Proc. Natl Acad. Sci. USA Vol 110, 88368841. http://dx.doi.org/10.1073/pnas.1212562110
50. Parise C. V. , Knorre K. , Ernst M. O. (2014). "Natural auditory scene statistics shapes human spatial hearing", Proc. Natl Acad. Sci. USA Vol 111, 61046108. http://dx.doi.org/10.1073/pnas.1322705111
51. Phillips B. , Zhao H. (1993). "Predictors of assistive technology abandonment", Assist. Technol. Vol 5, 3645. http://dx.doi.org/10.1080/10400435.1993.10132205
52. Roffler S. , Butler R. (1967). "Localization of tonal stimuli in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12601266. http://dx.doi.org/10.1121/1.1910977
53. Roffler S. K. , Butler R. A. (1968). "Factors that influence the localization of sound in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12551259. http://dx.doi.org/10.1121/1.1910976
54. Schwarz M. W. , Cowan W. B. , Beatty J. C. (1987). "An experimental comparison of RGB, YIQ, LAB, HSV, and opponent color models", ACM Trans. Graph. Vol 6, 123158. http://dx.doi.org/10.1145/31336.31338
55. Simpson R. H. , Quinn M. , Ausubel D. P. (1956). "Synesthesia in children: association of colors with pure tone frequencies", J. Genet. Psychol. Vol 89, 95103. http://dx.doi.org/10.1080/00221325.1956.10532990
56. Spence C. (2011). "Crossmodal correspondences: a tutorial review", Atten. Percept. Psychophys. Vol 73, 971995. http://dx.doi.org/10.3758/s13414-010-0073-7
57. Spence C. , Deroy O. (2012). "Crossmodal correspondences: innate or learned?", i-Perception Vol 3, 316318. http://dx.doi.org/10.1068/i0526ic
58. Spence C. , Deroy O. (2013). "How automatic are crossmodal correspondences?", Consc. Cogn. Vol 22, 245260. http://dx.doi.org/10.1016/j.concog.2012.12.006
59. Spence C. , Parise C. V. (2012). "The cognitive neuroscience of crossmodal correspondences", i-Perception Vol 3, 410412. http://dx.doi.org/10.1068/i0540ic
60. Stiles N. R. , Zheng Y. , Shimojo S. (2015). "Length and orientation constancy learning in 2-dimensions with auditory sensory substitution: the importance of self-initiated movement", Front. Psychol. Vol 6, 842. DOI:. http://dx.doi.org/10.3389/fpsyg.2015.00842
61. Tan S. S. , Maul T. H. B. , Mennie N. R. (2013). "Measuring the performance of visual to auditory information conversion", PLoS One Vol 8, e63042. DOI:. http://dx.doi.org/10.1371/annotation/da7ad86d-a5dc-4b03-981b-e44a2392c67b
62. Tkalcic M. , Tasic J. F. (2003). Colour spaces: perceptual, historical and applicational background, in: The IEEE Region 8 EUROCON 2003, Computer as a Tool, Vol. 1, pp. 304–308.
63. Torralba A. (2009). "How many pixels make an image?", Vis. Neurosci. Vol 26, 123131. http://dx.doi.org/10.1017/S0952523808080930
64. Walker P. , Bremner G. , Mason U. , Spring J. , Mattock K. , Slater A. , Johnson S. (2010). "Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences", Psychol Sci. Vol 21, 2125. http://dx.doi.org/10.1177/0956797609354734
65. Ward J. , Huckstep B. , Tsakanikos E. (2006). "Sound–colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all?", Cortex Vol 42, 264280. http://dx.doi.org/10.1016/S0010-9452(08)70352-6
66. Wessel D. L. (1979). "Timbre space as a musical control structure", Comput. Music J. Vol 3, 4552. http://dx.doi.org/10.2307/3680283
67. WHO (2014). Visual impairment and blindness. Retrieved from http://www.who.int/mediacentre/factsheets/fs282/en/index.html.
68. Wrembel M. (2009). "On hearing colours — cross-modal associations in vowel perception in a non-synaesthetic population", Poznań Stud. Contemp. Linguist. Vol 45, 595612.
69. Wright T. , Ward J. (2013). "The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms", Q. J. Exp. Psychol. Vol 66, 16201638. http://dx.doi.org/10.1080/17470218.2012.754911
http://brill.metastore.ingenta.com/content/journals/10.1163/22134808-00002519
Loading

Article metrics loading...

/content/journals/10.1163/22134808-00002519
2016-02-19
2017-11-23

Sign-in

Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
     
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation