Cookies Policy

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.

Affiliations: 1: School of Psychology and Sackler Centre for Consciousness Science, University of Sussex, Brighton, UK

Loading data from figshare Loading data from figshare

Full text loading...


Data & Media loading...

1. Abboud S. , Hanassy S. , Levy-Tzedek S. , Maidenbaum S. , Amedi A. (2014). "EyeMusic: introducing a ‘visual’ colorful experience for the blind using auditory sensory substitution", Restor. Neurol. Neurosci. Vol 32, 247257.
2. Amedi A. , Chebat D. R. , Levy-Tzedek S. , Buchs G. , Maidenbaum S. (2014). "Returning sensory substitution to practical visual rehabilitation", Invest. Ophthalmol. Vis. Sci. Vol 55, 4146.
3. Ancuti C. , Ancuti C. , Bekaert P. (2009). ColEnViSon: color enhanced visual sonifier a polyphonic audio texture and salient scene analysis, in: VISAPP 2009: Proceedings of the 4th International Conference on Computer Vision Theory and Applications, Lisbon, Portugal, Vol. 2, pp. 566–572.
4. Bien N. , ten Oever S. , Goebel R. , Sack A. T. (2012). "The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study", Neuroimage Vol 59, 663672.
5. Blauert J. (1997). Spatial Hearing. The Psychophysics of Human Sound Localization. MIT Press, Cambridge, MA, USA.
6. Bologna G. , Deville B. , Pun T. , Vinckenbosch A. (2007). "Transforming 3D coloured pixels into musical instrument notes for vision substitution applications", EURASIP J. Image Video Proc. Vol 2007, 076204. DOI:.
7. Bologna G. , Deville B. , Pun T. (2010). Blind navigation along a sinuous path by means of the See ColOr interface, in: Proceedings of the 3rd International Work-Conference on the Interplay Between Natural and Artifical Computation: Part II: Bioinspired Applications in Artifical and Natural Computation, Galicia, Spain, pp. 235–243.
8. Brewer C. A. (1999). Color use guidelines for data representation, in: Proceedings of the Section on Statistical Graphics, American Statistical Association, pp. 55–60.
9. Brown D. , Macpherson T. , Ward J. (2011). "Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device", Perception Vol 40, 11201135.
10. Brown D. J. , Simpson A. J. , Proulx M. J. (2014). "Visual objects in the auditory system in sensory substitution: how much information do we need?", Multisens. Res. Vol 27, 337357.
11. Burch D. (2012). Development of a multiple contact haptic display with texture-enhanced graphics, PhD Thesis, Virginia Commonwealth University, Richmond, VA, USA. Retrieved from
12. Bush M. (1999). Alternative marking schemes for on-line multiple choice tests, in: 7th Annual Conference on the Teaching of Computing, Belfast, UK.
13. Capalbo Z. , Glenney B. (2009). Hearing color: radical pluralistic realism and SSDs, in: Proceedings of AP-CAP, October 2009, Tokyo, Japan, pp. 135–140.
14. Capelle C. , Trullemans C. , Arno P. , Veraart C. (1998). "A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution", IEEE Trans. Biomed. Eng. Vol 45, 12791293.
15. Collignon O. , Lassonde M. , Lepore F. , Bastien D. , Veraart C. (2007). "Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects", Cereb. Cortex Vol 17, 457465.
16. Deroy O. , Spence C. (2013). "Are we all born synaesthetic? Examining the neonatal synaesthesia hypothesis", Neurosci. Biobehav. Rev. Vol 37, 12401253.
17. Doel K. (2003). SoundView: sensing color images by kinesthetic audio, in: Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA.
18. Elli G. V. , Benetti S. , Collignon O. (2014). "Is there a future for sensory substitution outside academic laboratories?", Multisens. Res. Vol 27, 271291.
19. Else L. (2012). "Cyborg makes art using seventh sense", New Sci. Vol 215, 50.
20. Fairchild M. D. (2005). Color Appearance Models, 2nd edn. John Wiley and Sons, Chichester, UK.
21. Froese T. , McGann M. , Bigge W. , Spiers A. , Seth A. K. (2012). "The enactive torch: a new tool for the science of perception", IEEE Trans. Haptics Vol 5, 365375.
22. Giannakis K. (2001). Sound mosaics: a graphical user interface for sound synthesis based on audio–visual associations, PhD Thesis, Middlesex University, London, UK. Retrieved from
23. Grey J. (1977). "Multidimensional percept scaling of musical timbres", J. Acoust. Soc. Am. Vol 61, 12701277.
24. Hamilton-Fletcher G. , Ward J. (2013). "Representing colour through hearing and touch in sensory substitution devices", Multisens. Res. Vol 26, 503.
25. Hamilton-Fletcher G. , Ward J. (2014). What does red sound like? Establishing and applying aesthetics to sensory substitution, poster session presented at Synaesthesia in Perspective: Development, Networks, and Multisensory Processing, Hamburg, Germany. DOI:.
26. Hubbard T. L. (1996). "Synesthesia-like mappings of lightness, pitch, and melodic interval", Am. J. Psychol. Vol 109, 219238.
27. Hunt R. W. G. , Pointer M. R. (2011). Measuring Colour, 4th edn. John Wiley and Sons, Chichester, UK.
28. Huopaniemi J. , Zacharov N. , Karjalainen M. (1999). "Objective and subjective evaluation of head-related transfer function filter design", J. Audio Eng. Soc. Vol 47, 218239.
29. ISO (2003). Acoustics — normal equal-loudness-level contours, ISO 226:2003, International Organisation for Standardisation, Geneva, Switzerland.
30. Kahol K. , French J. , Bratton L. , Panchanathan S. (2006). Learning and perceiving colors haptically, in: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, OR, USA, pp. 173–180.
31. Levy-Tzedek S. , Riemer D. , Amedi A. (2014). "Color improves ‘visual’ acuity via sound", Front. Neurosci. Vol 8, 358. DOI:.
32. Lewkowicz D. , Turkewitz G. (1980). "Cross-modal equivalence in early infancy: auditory–visual intensity matching", Dev. Psychol. Vol 6, 597607.
33. Ludwig V. U. , Adachi I. , Matsuzawa T. (2011). "Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans", Proc. Natl Acad. Sci. Vol 108, 2066120665.
34. Maidenbaum S. , Amedi A. (2015). Non-visual virtual interaction: can sensory substitution generically increase the accessibility of graphical virtual reality to the blind?, in: IEEE VR 3rd International Workshop on Virtual and Augmented Assistive Technology (VAAT), Arles, France, pp. 15–17.
35. Maidenbaum S. , Abboud S. , Amedi A. (2014a). "Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation", Neurosci. Biobehav. Rev. Vol 41, 315.
36. Maidenbaum S. , Hanassy S. , Abboud S. , Buchs G. , Chebat D. R. , Levy-Tzedek S. , Amedi A. (2014b). "The ‘EyeCane’, a new electronic travel aid for the blind: technology, behavior and swift learning", Restor. Neurol. Neurosci. Vol 32, 813824.
37. Marks L. E. (1974). "On associations of light and sound: the mediation of brightness, pitch, and loudness", Am. J. Psychol. Vol 87, 173188.
38. Marks L. E. (1975). "On colored-hearing synesthesia: cross-modal translations of sensory dimensions", Psychol. Bull. Vol 82, 303331.
39. Marks L. E. (1987). "On cross-modal similarity: auditory–visual interactions in speeded discrimination", J. Exp. Psychol. Hum. Percept. Perform. Vol 13, 384394.
40. Martino G. , Marks L. E. (1999). "Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis", Perception Vol 28, 903923.
41. Meers S. , Ward K. (2004). A vision system for providing 3D perception of the environment via transcutaneous electro-neural stimulation, in: Proceedings of the Information Visualisation, 8th International Conference, pp. 546–552.
42. Meers S. , Ward K. (2005). A substitute vision system for providing 3D perception and GPS navigation via electro-tactile stimulation, in: Proceedings of the International Conference on Sensing Technology, Palmerston North, New Zealand, pp. 551–556.
43. Meijer P. (1992). "An experimental system for auditory image representations", IEEE Trans. Biomed. Eng. Vol 39, 112121.
44. Melara R. D. (1989). "Dimensional interaction between color and pitch", J. Exp. Psychol. Hum. Percept. Perform. Vol 15, 6979.
45. Melara R. D. , O’Brien T. (1987). "Interaction between synesthetically corresponding dimensions", J. Exp. Psychol. Gen. Vol 116, 323336.
46. Miyahara T. , Koda A. , Sekiguchi R. , Amemiya T. (2012). "A psychological experiment on the correspondence between colors and voiced vowels in non-synesthetes", Kansei Eng. Int. J. Vol 11, 2734.
47. Moos A. , Smith R. , Miller S. R. , Simmons D. R. (2014). "Cross-modal associations in synaesthesia: vowel colours in the ear of the beholder", i-Perception Vol 5, 132142.
48. Orlandatou K. (2012). The role of pitch and timbre in the synaesthetic experience, in: Proceedings of the 12th International Conference on Music Perception and Cognition and the 8th Triennial Conference of the European Society for the Cognitive Sciences of Music, Thessaloniki, Greece, pp. 751–758.
49. Palmer S. E. , Schloss K. B. , Xu Z. , Prado-León L. R. (2013). "Music–color associations are mediated by emotion", Proc. Natl Acad. Sci. USA Vol 110, 88368841.
50. Parise C. V. , Knorre K. , Ernst M. O. (2014). "Natural auditory scene statistics shapes human spatial hearing", Proc. Natl Acad. Sci. USA Vol 111, 61046108.
51. Phillips B. , Zhao H. (1993). "Predictors of assistive technology abandonment", Assist. Technol. Vol 5, 3645.
52. Roffler S. , Butler R. (1967). "Localization of tonal stimuli in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12601266.
53. Roffler S. K. , Butler R. A. (1968). "Factors that influence the localization of sound in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12551259.
54. Schwarz M. W. , Cowan W. B. , Beatty J. C. (1987). "An experimental comparison of RGB, YIQ, LAB, HSV, and opponent color models", ACM Trans. Graph. Vol 6, 123158.
55. Simpson R. H. , Quinn M. , Ausubel D. P. (1956). "Synesthesia in children: association of colors with pure tone frequencies", J. Genet. Psychol. Vol 89, 95103.
56. Spence C. (2011). "Crossmodal correspondences: a tutorial review", Atten. Percept. Psychophys. Vol 73, 971995.
57. Spence C. , Deroy O. (2012). "Crossmodal correspondences: innate or learned?", i-Perception Vol 3, 316318.
58. Spence C. , Deroy O. (2013). "How automatic are crossmodal correspondences?", Consc. Cogn. Vol 22, 245260.
59. Spence C. , Parise C. V. (2012). "The cognitive neuroscience of crossmodal correspondences", i-Perception Vol 3, 410412.
60. Stiles N. R. , Zheng Y. , Shimojo S. (2015). "Length and orientation constancy learning in 2-dimensions with auditory sensory substitution: the importance of self-initiated movement", Front. Psychol. Vol 6, 842. DOI:.
61. Tan S. S. , Maul T. H. B. , Mennie N. R. (2013). "Measuring the performance of visual to auditory information conversion", PLoS One Vol 8, e63042. DOI:.
62. Tkalcic M. , Tasic J. F. (2003). Colour spaces: perceptual, historical and applicational background, in: The IEEE Region 8 EUROCON 2003, Computer as a Tool, Vol. 1, pp. 304–308.
63. Torralba A. (2009). "How many pixels make an image?", Vis. Neurosci. Vol 26, 123131.
64. Walker P. , Bremner G. , Mason U. , Spring J. , Mattock K. , Slater A. , Johnson S. (2010). "Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences", Psychol Sci. Vol 21, 2125.
65. Ward J. , Huckstep B. , Tsakanikos E. (2006). "Sound–colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all?", Cortex Vol 42, 264280.
66. Wessel D. L. (1979). "Timbre space as a musical control structure", Comput. Music J. Vol 3, 4552.
67. WHO (2014). Visual impairment and blindness. Retrieved from
68. Wrembel M. (2009). "On hearing colours — cross-modal associations in vowel perception in a non-synaesthetic population", Poznań Stud. Contemp. Linguist. Vol 45, 595612.
69. Wright T. , Ward J. (2013). "The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms", Q. J. Exp. Psychol. Vol 66, 16201638.

Article metrics loading...



Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation