Cookies Policy

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Representing Colour Through Hearing and Touch in Sensory Substitution Devices

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Visual sensory substitution devices (SSDs) allow visually-deprived individuals to navigate and recognise the ‘visual world’; SSDs also provide opportunities for psychologists to study modality-independent theories of perception. At present most research has focused on encoding greyscale vision. However at the low spatial resolutions received by SSD users, colour information enhances object-ground segmentation, and provides more stable cues for scene and object recognition. Many attempts have been made to encode colour information in tactile or auditory modalities, but many of these studies exist in isolation. This review brings together a wide variety of tactile and auditory approaches to representing colour. We examine how each device constructs ‘colour’ relative to veridical human colour perception and report previous experiments using these devices. Theoretical approaches to encoding and transferring colour information through sound or touch are discussed for future devices, covering alternative stimulation approaches, perceptually distinct dimensions and intuitive cross-modal correspondences.

Affiliations: 1: School of Psychology and Sackler Centre for Consciousness Science, University of Sussex, UK


Full text loading...


Data & Media loading...

1. Ancuti C. , Ancuti C. , Bekaert P. ( 2009). ColEnViSon: Color Enhanced Visual Sonifier A polyphonic audio texture and salient scene analysis, in: VISAPP 2009: Proceedings of the 4th International Conference on Computer Vision Theory and Applications, Lisbon, Portugal, Vol. 2, pp. 566–572.
2. Bach-y-Rita P. , Kaczmarek K. , Tyler M. , Garcia-Lara J. ( 1998). "Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note", J. Rehabil. Res. Dev. Vol 35, 427430.
3. Barrass S. ( 1997). Auditory information design, Doctoral dissertation, The Australian National University. Retrieved August 16th, 2012, from
4. Bologna G. , Deville B. , Pun T. , Vinckenbosch A. ( 2007). "Transforming 3D coloured pixels into musical instrument notes for vision substitution applications", EURASIP J. Image Video Process, Hong Kong, China. DOI:
5. Bologna G. , Deville B. , Vinckenbosch M. , Pun T. ( 2008). A perceptual interface for vision substitution in a color matching experiment, in: 2008 IEEE International Joint Conference on Neural Networks, Vols 1–8, pp. 1621–1628.
6. Bologna G. , Deville B. , Pun T. ( 2010). Blind navigation along a sinuous path by means of the See ColOr interface, in: Proceedings of the 3rd International Work-Conference on the Interplay between Natural and Artifical Computation: Part II: Bioinspired Applications in Artifical and Natural Computation, Galicia, Spain, pp. 235–243.
7. Bornstein M. ( 1987). "Perceptual categories in vision audition", in: Categorical Perception, Harnad S. (Ed.), pp.  287300. Cambridge University Press, Cambridge, UK.
8. Burch D. ( 2012). Development of a multiple contact haptic display with texture-enhanced graphics, Doctoral Dissertation, Virginia Commonwealth University. Retrieved August 16th, 2012, from
9. Burch D. , Pawluk D. ( 2009). A cheap, portable haptic device for a method to relay 2-D texture-enriched graphical information to individuals who are visually impaired, in: Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility, Pittsburgh, PA, USA, pp. 215–216.
10. Capalbo Z. , Glenney B. ( 2009). Hearing color: radical pluralistic realism and SSDs, in: Proceedings of AP-CAP October 2009, Tokyo, Japan, pp. 135–141.
11. Cappelletti L. , Ferri M. , Nicoletti G. ( 1998). Vibrotactile colour rendering for the visually impaired within the VIDET project, in: Proc. SPIE 3524, Telemanipulator and Telepresence Technologies V, M. R. Stein (Ed.), pp. 92–96.
12. Chebat D.-R. , Rainville C. , Kupers R. , Ptito M. ( 2007). "Tactile–‘visual’ acuity of the tongue in early blind individuals", Neuroreport Vol 18, 19011904.
13. Chebat D.-R. , Schneider F. , Kupers R. , Ptito M. ( 2011). "Navigation with a sensory substitution device in congenitally blind individuals", Neuroreport Vol 22, 342347.
14. Crisman J. , Thorpe C. ( 1993). "SCARF: a color vision system that tracks roads and intersections", IEEE Trans. Rob. Autom. Vol 9, 4958.
15. Doel K. ( 2003). SoundView: sensing color images by kinesthetic audio, in: Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, pp. 303–306.
16. Doel K. , Smilek D. , Bodnar A. , Chita C. , Corbett R. , Nekrasovski D. , McGrenere J. ( 2004). Geometric shape detection with SoundView, in: Proceedings of ICAD 04 — Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, pp. 1–8.
17. Dotov D. G. , Nie L. , Chemero A. ( 2010). "A demonstration of the transition from ready-to-hand to unready-to-hand", Plos ONE Vol 5, e9433. DOI:
18. Ebert J. ( 2005). "Scientists with disabilities: access all areas", Nature Vol 435, 552554.
19. Else L. ( 2012). "Cyborg makes art using seventh sense", New Sci. Vol 215, 50.
20. Engel S. ( 1999). "Using neuroimaging to measure mental representations: finding color-opponent neurons in visual cortex", Curr. Dir. Psychol. Sci. Vol 8, 2327.
21. Goffaux V. , Jacques C. , Mouraux A. , Oliva A. , Schyns P. G. , Rossion B. ( 2005). "Diagnostic colours contribute to the early stages of scene categorization: behavioural and neurophysiological evidence", Vis. Cogn. Vol 12, 878892.
22. Golledge R. , Marston J. , Loomis J. , Klatzky R. ( 2004). "Stated preferences for components of a personal guidance system for nonvisual navigation", J. Vis. Impair. Blind. Vol 98, 135147.
23. Grantham D. , Hornsby B. , Erpenbeck E. ( 2003). "Auditory spatial resolution in horizontal, vertical, and diagonal planes", J. Acoust. Soc. Am. Vol 114, 10091022.
24. Grey J. ( 1975). Exploration of musical timbre, Doctoral Dissertation, Stanford University.
25. Grey J. ( 1977). "Multidimensional percept scaling of musical timbres", J. Acoust. Soc. Am. Vol 61, 12701277.
26. Haigh A. , Brown D. , Meijer P. , Proulx M. ( 2013). "How well do you see what you hear? The acuity of visual-to-auditory sensory substitution", Front. Psychol. Vol 4, 330.
27. Hansen T. , Walter S. , Gegenfurtner K. ( 2007). "Effects of spatial and temporal context on color categories and color constancy", J. Vis. Vol 7, 115.
28. Harnad S. ( 1987). "Psychophysical and cognitive aspects of categorical perception: a critical overview", in: Categorical Perception: The Groundwork of Cognition, Harnad S. (Ed.), pp.  535565. Cambridge University Press, Cambridge, UK.
29. Harris J. , Arabzadeh E. , Fairhall A. , Benito C. , Diamond M. ( 2006). "Factors affecting frequency discrimination of vibrotactile stimuli: implications for cortical encoding", PLoS ONE Vol 1, e100. DOI:
30. Hollins M. , Goble A. , Whitsel B. , Tommerdahl M. ( 1990). "Time course and action spectrum of vibrotactile adaptation", Somatosens. Mot. Res. Vol 7, 205221.
31. Hribar V. , Pawluk D. ( 2011). A tactile-thermal display for haptic exploration of virtual paintings, in: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, Dundee, Scotland, UK, pp. 221–222.
32. Hubbard T. ( 1996). "Synesthesia-like mappings of lightness, pitch, and melodic interval", Am. J. Psychol. Vol 109, 219238.
33. Hurvich L. , Jameson D. ( 1955). "Some quantitative aspects of an opponent-colors theory: II. Brightness, saturation and hue in normal and dichromatic vision", J. Opt. Soc. Am. Vol 45, 602616.
34. Hurvich L. , Jameson D. ( 1957). "An opponent-process theory of color vision", Psychol. Rev. Vol 64, 384404.
35. Jewanski J. ( 2010). Color-tone analogies: a systematic presentation of the principles of correspondence, in: Audiovisuology: Compendium. A Multidisciplinary Survey of Audiovisual Culture, D. Daniels, S. Naumann and J. Thoben (Eds), König, Cologne, Germany, pp. 339–347.
36. Kaczmarek K. , Webster J. , Bach-y-Rita P. , Tompkins W. ( 1991). "Electrotactile and vibrotactile displays for sensory substitution systems", IEEE Trans. Biomed. Eng. Vol 38, 116.
37. Kahol K. , French J. , Bratton L. , Panchanathan S. ( 2006). Learning and perceiving colors haptically, in: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, OR, USA, pp. 173–180.
38. Kajimoto H. , Kawakami N. , Tachi S. ( 2004). Electro-tactile display with tactile primary color approach, in: International Conference on Intelligent Robots and Systems, Sendai, Japan. Retrieved August 16th, 2012 from
39. Kelly K. , Judd D. ( 1976). Color; Universal Language and Dictionary of Names. Lightning Source UK Ltd, Milton Keynes, UK.
40. Khan A. , Moideen F. , Lopez J. , Khoo W. , Zhu Z. ( 2012). KinDectect: kinect detecting objects, in: Proceedings of the 13th International Conference on Computer Helping People with Special Needs — Volume Part II, Linz, Austria, pp. 588–595.
41. Kokjer K. ( 1987). "The information capacity of the human fingertip", IEEE Trans. Syst. Man. Cybern. Vol 17, 100102.
42. Levy-Tzedek S. , Hanassy S. , Abboud S. , Maidenbaum S. , Amedi A. ( 2012a). "Fast, accurate reaching movements with a visual-to-auditory sensory substitution device", Restor. Neurol. Neurosci. Vol 31, 111.
43. Levy-Tzedek S. , Novick I. , Arbel R. , Abboud S. , Maidenbaum S. , Vaadia E. , Amedi A. ( 2012b). "Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution", Sci. Rep. Vol 2, article 949. DOI: .
44. Lewkowicz D. , Turkewitz G. ( 1980). "Cross-modal equivalence in early infancy: auditory–visual intensity matching", Dev. Psychol. Vol 6, 597607.
45. Ludwig V. U. , Simner J. ( 2013). "What colour does that feel? Tactile–visual mapping and the development of cross-modality", Cortex Vol 49, 10891099.
46. Mann S. , Huang J. , Janzen R. , Lo R. , Rampersad V. , Chen A. , Doha T. ( 2011). Blind navigation with a wearable range camera and vibrotactile helmet, in: Proceedings of the 19th ACM International Conference on Multimedia, Scotsdale, AZ, USA, pp. 1325–1328.
47. Marks L. ( 1974). "On associations of light and sound: the mediation of brightness, pitch, and loudness", American J. Psychol. Vol 87, 173188.
48. Marks L. ( 1987). "On cross-modal similarity: auditory–visual interactions in speeded discrimination", J. Exp. Psychol. Human Percept. Perform. Vol 13, 384394.
49. Martino G. , Marks L. ( 2000). "Cross-modal interaction between vision and touch: the role of synesthetic correspondence", Perception Vol 29, 745754.
50. Matteau I. , Kupers R. , Ricciardi E. , Pietrini P. , Ptito M. ( 2010). "Beyond visual, aural and haptic movement perception: hMT+ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals", Brain Res. Bull. Vol 82, 264270.
51. McMorrow G. , Wang X. , Whelan P. ( 1997). A color-to-speech sensory substitution device for the visually impaired, in: Machine Vision Applications, Architectures, and Systems Integration VI 1997, Pittsburgh, PA, USA, pp. 272–281.
52. Meers S. , Ward K. ( 2004). A vision system for providing 3D perception of the environment via transcutaneous electro-neural stimulation, in: Proceedings of the Information Visualisation, 8th International Conference, London, UK, pp. 546–552.
53. Meers S. , Ward K. ( 2005). A substitute vision system for providing 3d perception and gps navigation via electro-tactile stimulation, in: Proceedings of the International Conference on Sensing Technology, Palmerston North, New Zealand, pp. 551–556.
54. Meers S. , Ward K. ( 2007). Substitute three-dimensional perception using depth and colour sensors. Retrieved August 16th, 2012, from
55. Meijer P. ( 1992). "An experimental system for auditory image representations", IEEE Trans. Biomed. Eng. Vol 39, 112121.
56. Melara R. , O’Brien T. ( 1987). "Interaction between synesthetically corresponding dimensions", J. Exp. Psychol. Gen. Vol 116, 323336.
57. Morgan G. , Goodson F. , Jones T. ( 1975). "Age differences in the associations between felt temperatures and color choices", Am. J. Psychol. Vol 88, 125130.
58. Morley J. , Rowe M. ( 1990). "Perceived pitch of vibrotactile stimuli — effects of vibration amplitude, and implications for vibration frequency coding", J. Physiol. — London Vol 431, 403416.
59. O’Regan K. , Noë A. ( 2001). "A sensorimotor account of vision and visual consciousness", Behav. Brain Sci. Vol 24, 9391031.
60. Orwell J. , Remagnino P. M. , Jones G. A. ( 2001). "Optimal color quantization for real-time object recognition", Real-Time Imaging Vol 7, 401414.
61. Ou L. , Luo M. , Sun P. , Hu N. , Chen H. , Guan S. , Woodcock A. , Caivano J. , Huertas R. , Tremeau A. , Billger M. , Izaban H. , Richter K. ( 2010). "A cross-cultural comparison of colour emotion for two-colour combinations", Color Res. Appl. Vol 37, 2343.
62. Regier T. , Kay P. , Cook R. ( 2005). "Focal colors are universal after all", Proc. Natl Acad. Sci. USA Vol 102, 83868391.
63. Rivest J. , Cavanagh P. ( 1996). "Localizing contours defined by more than one attribute", Vis. Res. Vol 36, 5366.
64. Roffler S. , Butler R. ( 1967). "Localization of tonal stimuli in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12601266.
65. Rossi J. , Perales F. , Varona J. , Roca M. ( 2009). Col.diesis: transforming colour into melody and implementing the result in a colour sensor device, in: Proceedings for the 2nd International Conference in Visualisation, Barcelona, Spain, pp. 30–35.
66. Roth P. , Richoz D. , Petrucci L. , Pun T. ( 2001). An audio-haptic tool for non-visual image representation, in: International Symposium on Signal Processing and Its Applications (ISSPA), Kuala Lumpur, Malaysia, pp. 64–67.
67. Rousselet G. , Joubert O. , Fabre-Thorpe M. ( 2005). "How long to get to the “gist” of real-world natural scenes?" Vis. Cogn. Vol 12, 852877.
68. Roy E. , Hollins M. ( 1998). "A ratio code for vibrotactile pitch", Somatosens. Mot. Res. Vol 15, 134145.
69. Sampaio E. , Maris S. , Bachy-y-Rita P. ( 2001). "Brain plasticity: “visual” acuity of blind persons via the tongue", Brain Res. Vol 908, 204207.
70. Schwerdt H. N. , Tapson J. , Etienne-Cummings R. ( 2009). A color detection glove with haptic feedback for the visually disabled, in: 43rd Annual Conference on Information Science and Systems, 2009, Baltimore, MD, USA, pp. 681–686.
71. Segond H. , Weiss D. , Sampaio E. ( 2005). "Human spatial navigation via a visuo-tactile sensory substitution system", Perception Vol 34, 12311249.
72. Shepard R. ( 1964). "Circularity in judgments of relative pitch", J. Acoust. Soc. Am. Vol 36, 23462353.
73. Sjöström C. , Rassmus-Gröhn K. ( 1999). "The sense of touch provides new computer interaction techniques for disabled people", Technol. Disabil. Vol 10, 4552.
74. Striem-Amit E. , Guendelman M. , Amedi A. ( 2012). "‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution", Plos ONE Vol 7, e33136. DOI:
75. Tapson J. , Gurari N. , Diaz J. , Chicca E. , Sander D. , Pouliquen P. , Etienne-Cummings R. ( 2008). The feeling of color: a haptic feedback device for the visually disabled, in: Proceedings for the IEEE Biomedical Circuits and Systems Conference, 2008. BioCAS, Seattle, WA, USA, pp. 281–384.
76. Taylor B. ( 1977). "Dimensional interactions in vibrotactile information-processing", Percept. Psychophys. Vol 21, 477481.
77. Torralba A. ( 2009). "How many pixels make an image?" Vis. Neurosci. Vol 26, 123131.
78. Vallbo A. , Johansson R. ( 1984). "Properties of cutaneous mechanoreceptors in the human hand related to touch sensation", Hum. Neurobiol. Vol 3, 314.
79. Verrillo R. , Fraioli A. , Smith R. ( 1969). "Sensation magnitude of vibrotactile stimuli", Percept. Psychophys. Vol 6, 366372.
80. Visell Y. ( 2009). "Tactile sensory substitution: models for enaction in HCI", Interact. Comput. Vol 21, 3853.
81. Von Bismarck G. ( 1974). "Timbre of steady tones: a factorial investigation of its verbal attributes", Acustica Vol 30, 146159.
82. Walker P. , Bremner G. , Mason U. , Spring J. , Mattock K. , Slater A. , Johnson S. ( 2010). "Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences", Psychol. Sci. Vol 21, 2125.
83. Wang D. , Brown G. ( 2006). Computational Auditory Scene Analysis: Principles, Algorithms, and Applications. Wiley-Blackwell, New York, NY, USA.
84. Ward J. , Wright T. (in press). "Sensory substitution as an artificially acquired synaesthesia", Neurosci. Biobehav. Rev.DOI:
85. Ward J. , Huckstep B. , Tsakanikos E. ( 2006). "Sound–colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all?" Cortex Vol 42, 264280.
86. Way T. P. , Barner K. E. ( 1997a). "Automatic visual to tactile translation. Part I: human factors, access methods, and image manipulation", IEEE Trans. Rehabil. Eng. Vol 5, 8194.
87. Way T. P. , Barner K. E. ( 1997b). "Automatic visual to tactile translation. Part II: evaluation of the TACTile Image Creation System", IEEE Trans. Rehabil. Eng. Vol 5, 95105.
88. Wessel D. ( 1985). "Timbre space as a musical control structure", in: Foundations of Computer Music, Roads C. , Strawn J. (Eds), pp.  4552. MIT Press, Cambridge, MA, USA.
89. WHO( 2012). Visual impairment and blindness. Retrieved August 16th, 2012, from
90. Yang G.-H. , Kyung K.-U. , Srinivasan M. A. , Kwon D.-S. ( 2006). Quantitative tactile display device with pin-array type tactile feedback and thermal feedback, in: 2006 IEEE International Conference on Robotics and Automation (ICRA), Vols 1–10, Orlando, FL, USA, pp. 3917–3922.
91. Zacharakis A. , Pastiadis K. , Papadelis G. , Reiss J. ( 2011). An investigation of musical timbre: uncovering salient semantic descriptors and perceptual dimensions, in: Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), Miami, FL, USA, pp. 807–812.

Article metrics loading...



Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation