Cookies Policy
X

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Representing Colour Through Hearing and Touch in Sensory Substitution Devices

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Visual sensory substitution devices (SSDs) allow visually-deprived individuals to navigate and recognise the ‘visual world’; SSDs also provide opportunities for psychologists to study modality-independent theories of perception. At present most research has focused on encoding greyscale vision. However at the low spatial resolutions received by SSD users, colour information enhances object-ground segmentation, and provides more stable cues for scene and object recognition. Many attempts have been made to encode colour information in tactile or auditory modalities, but many of these studies exist in isolation. This review brings together a wide variety of tactile and auditory approaches to representing colour. We examine how each device constructs ‘colour’ relative to veridical human colour perception and report previous experiments using these devices. Theoretical approaches to encoding and transferring colour information through sound or touch are discussed for future devices, covering alternative stimulation approaches, perceptually distinct dimensions and intuitive cross-modal correspondences.

Affiliations: 1: School of Psychology and Sackler Centre for Consciousness Science, University of Sussex, UK

Loading

Full text loading...

/content/journals/10.1163/22134808-00002434
Loading

Data & Media loading...

1. Ancuti C. , Ancuti C. , Bekaert P. ( 2009). ColEnViSon: Color Enhanced Visual Sonifier A polyphonic audio texture and salient scene analysis, in: VISAPP 2009: Proceedings of the 4th International Conference on Computer Vision Theory and Applications, Lisbon, Portugal, Vol. 2, pp. 566–572.
2. Bach-y-Rita P. , Kaczmarek K. , Tyler M. , Garcia-Lara J. ( 1998). "Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note", J. Rehabil. Res. Dev. Vol 35, 427430.
3. Barrass S. ( 1997). Auditory information design, Doctoral dissertation, The Australian National University. Retrieved August 16th, 2012, from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.127.6228&rep=rep1&type=pdf
4. Bologna G. , Deville B. , Pun T. , Vinckenbosch A. ( 2007). "Transforming 3D coloured pixels into musical instrument notes for vision substitution applications", EURASIP J. Image Video Process, Hong Kong, China. DOI:
5. Bologna G. , Deville B. , Vinckenbosch M. , Pun T. ( 2008). A perceptual interface for vision substitution in a color matching experiment, in: 2008 IEEE International Joint Conference on Neural Networks, Vols 1–8, pp. 1621–1628.
6. Bologna G. , Deville B. , Pun T. ( 2010). Blind navigation along a sinuous path by means of the See ColOr interface, in: Proceedings of the 3rd International Work-Conference on the Interplay between Natural and Artifical Computation: Part II: Bioinspired Applications in Artifical and Natural Computation, Galicia, Spain, pp. 235–243.
7. Bornstein M. ( 1987). "Perceptual categories in vision audition", in: Categorical Perception, Harnad S. (Ed.), pp.  287300. Cambridge University Press, Cambridge, UK.
8. Burch D. ( 2012). Development of a multiple contact haptic display with texture-enhanced graphics, Doctoral Dissertation, Virginia Commonwealth University. Retrieved August 16th, 2012, from https://digarchive.library.vcu.edu/handle/10156/3836
9. Burch D. , Pawluk D. ( 2009). A cheap, portable haptic device for a method to relay 2-D texture-enriched graphical information to individuals who are visually impaired, in: Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility, Pittsburgh, PA, USA, pp. 215–216.
10. Capalbo Z. , Glenney B. ( 2009). Hearing color: radical pluralistic realism and SSDs, in: Proceedings of AP-CAP October 2009, Tokyo, Japan, pp. 135–141.
11. Cappelletti L. , Ferri M. , Nicoletti G. ( 1998). Vibrotactile colour rendering for the visually impaired within the VIDET project, in: Proc. SPIE 3524, Telemanipulator and Telepresence Technologies V, M. R. Stein (Ed.), pp. 92–96.
12. Chebat D.-R. , Rainville C. , Kupers R. , Ptito M. ( 2007). "Tactile–‘visual’ acuity of the tongue in early blind individuals", Neuroreport Vol 18, 19011904. http://dx.doi.org/10.1097/WNR.0b013e3282f2a63
13. Chebat D.-R. , Schneider F. , Kupers R. , Ptito M. ( 2011). "Navigation with a sensory substitution device in congenitally blind individuals", Neuroreport Vol 22, 342347. http://dx.doi.org/10.1097/WNR.0b013e3283462def
14. Crisman J. , Thorpe C. ( 1993). "SCARF: a color vision system that tracks roads and intersections", IEEE Trans. Rob. Autom. Vol 9, 4958. http://dx.doi.org/10.1109/70.210794
15. Doel K. ( 2003). SoundView: sensing color images by kinesthetic audio, in: Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, pp. 303–306.
16. Doel K. , Smilek D. , Bodnar A. , Chita C. , Corbett R. , Nekrasovski D. , McGrenere J. ( 2004). Geometric shape detection with SoundView, in: Proceedings of ICAD 04 — Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, pp. 1–8.
17. Dotov D. G. , Nie L. , Chemero A. ( 2010). "A demonstration of the transition from ready-to-hand to unready-to-hand", Plos ONE Vol 5, e9433. DOI: http://dx.doi.org/10.1371/journal.pone.0009433
18. Ebert J. ( 2005). "Scientists with disabilities: access all areas", Nature Vol 435, 552554. http://dx.doi.org/10.1038/435552a
19. Else L. ( 2012). "Cyborg makes art using seventh sense", New Sci. Vol 215, 50. http://dx.doi.org/10.1016/S0262-4079(12)62082-X
20. Engel S. ( 1999). "Using neuroimaging to measure mental representations: finding color-opponent neurons in visual cortex", Curr. Dir. Psychol. Sci. Vol 8, 2327. http://dx.doi.org/10.1111/1467-8721.00006
21. Goffaux V. , Jacques C. , Mouraux A. , Oliva A. , Schyns P. G. , Rossion B. ( 2005). "Diagnostic colours contribute to the early stages of scene categorization: behavioural and neurophysiological evidence", Vis. Cogn. Vol 12, 878892. http://dx.doi.org/10.1080/13506280444000562
22. Golledge R. , Marston J. , Loomis J. , Klatzky R. ( 2004). "Stated preferences for components of a personal guidance system for nonvisual navigation", J. Vis. Impair. Blind. Vol 98, 135147.
23. Grantham D. , Hornsby B. , Erpenbeck E. ( 2003). "Auditory spatial resolution in horizontal, vertical, and diagonal planes", J. Acoust. Soc. Am. Vol 114, 10091022. http://dx.doi.org/10.1121/1.1590970
24. Grey J. ( 1975). Exploration of musical timbre, Doctoral Dissertation, Stanford University.
25. Grey J. ( 1977). "Multidimensional percept scaling of musical timbres", J. Acoust. Soc. Am. Vol 61, 12701277. http://dx.doi.org/10.1121/1.381428
26. Haigh A. , Brown D. , Meijer P. , Proulx M. ( 2013). "How well do you see what you hear? The acuity of visual-to-auditory sensory substitution", Front. Psychol. Vol 4, 330. http://dx.doi.org/10.3389/fpsyg.2013.00330
27. Hansen T. , Walter S. , Gegenfurtner K. ( 2007). "Effects of spatial and temporal context on color categories and color constancy", J. Vis. Vol 7, 115.
28. Harnad S. ( 1987). "Psychophysical and cognitive aspects of categorical perception: a critical overview", in: Categorical Perception: The Groundwork of Cognition, Harnad S. (Ed.), pp.  535565. Cambridge University Press, Cambridge, UK.
29. Harris J. , Arabzadeh E. , Fairhall A. , Benito C. , Diamond M. ( 2006). "Factors affecting frequency discrimination of vibrotactile stimuli: implications for cortical encoding", PLoS ONE Vol 1, e100. DOI: http://dx.doi.org/10.1371/journal.pone.0000100
30. Hollins M. , Goble A. , Whitsel B. , Tommerdahl M. ( 1990). "Time course and action spectrum of vibrotactile adaptation", Somatosens. Mot. Res. Vol 7, 205221. http://dx.doi.org/10.3109/08990229009144707
31. Hribar V. , Pawluk D. ( 2011). A tactile-thermal display for haptic exploration of virtual paintings, in: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, Dundee, Scotland, UK, pp. 221–222.
32. Hubbard T. ( 1996). "Synesthesia-like mappings of lightness, pitch, and melodic interval", Am. J. Psychol. Vol 109, 219238. http://dx.doi.org/10.2307/1423274
33. Hurvich L. , Jameson D. ( 1955). "Some quantitative aspects of an opponent-colors theory: II. Brightness, saturation and hue in normal and dichromatic vision", J. Opt. Soc. Am. Vol 45, 602616. http://dx.doi.org/10.1364/JOSA.45.000602
34. Hurvich L. , Jameson D. ( 1957). "An opponent-process theory of color vision", Psychol. Rev. Vol 64, 384404. http://dx.doi.org/10.1037/h0041403
35. Jewanski J. ( 2010). Color-tone analogies: a systematic presentation of the principles of correspondence, in: Audiovisuology: Compendium. A Multidisciplinary Survey of Audiovisual Culture, D. Daniels, S. Naumann and J. Thoben (Eds), König, Cologne, Germany, pp. 339–347.
36. Kaczmarek K. , Webster J. , Bach-y-Rita P. , Tompkins W. ( 1991). "Electrotactile and vibrotactile displays for sensory substitution systems", IEEE Trans. Biomed. Eng. Vol 38, 116. http://dx.doi.org/10.1109/10.68204
37. Kahol K. , French J. , Bratton L. , Panchanathan S. ( 2006). Learning and perceiving colors haptically, in: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Portland, OR, USA, pp. 173–180.
38. Kajimoto H. , Kawakami N. , Tachi S. ( 2004). Electro-tactile display with tactile primary color approach, in: International Conference on Intelligent Robots and Systems, Sendai, Japan. Retrieved August 16th, 2012 from http://midas.inrf.uci.edu/groups/bats/uploads/Xsense/Kajimoto_IROS_2004.pdf
39. Kelly K. , Judd D. ( 1976). Color; Universal Language and Dictionary of Names. Lightning Source UK Ltd, Milton Keynes, UK.
40. Khan A. , Moideen F. , Lopez J. , Khoo W. , Zhu Z. ( 2012). KinDectect: kinect detecting objects, in: Proceedings of the 13th International Conference on Computer Helping People with Special Needs — Volume Part II, Linz, Austria, pp. 588–595.
41. Kokjer K. ( 1987). "The information capacity of the human fingertip", IEEE Trans. Syst. Man. Cybern. Vol 17, 100102. http://dx.doi.org/10.1109/TSMC.1987.289337
42. Levy-Tzedek S. , Hanassy S. , Abboud S. , Maidenbaum S. , Amedi A. ( 2012a). "Fast, accurate reaching movements with a visual-to-auditory sensory substitution device", Restor. Neurol. Neurosci. Vol 31, 111.
43. Levy-Tzedek S. , Novick I. , Arbel R. , Abboud S. , Maidenbaum S. , Vaadia E. , Amedi A. ( 2012b). "Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution", Sci. Rep. Vol 2, article 949. DOI: . http://dx.doi.org/10.1038/srep00949
44. Lewkowicz D. , Turkewitz G. ( 1980). "Cross-modal equivalence in early infancy: auditory–visual intensity matching", Dev. Psychol. Vol 6, 597607. http://dx.doi.org/10.1037/0012-1649.16.6.597
45. Ludwig V. U. , Simner J. ( 2013). "What colour does that feel? Tactile–visual mapping and the development of cross-modality", Cortex Vol 49, 10891099. http://dx.doi.org/10.1016/j.cortex.2012.04.004
46. Mann S. , Huang J. , Janzen R. , Lo R. , Rampersad V. , Chen A. , Doha T. ( 2011). Blind navigation with a wearable range camera and vibrotactile helmet, in: Proceedings of the 19th ACM International Conference on Multimedia, Scotsdale, AZ, USA, pp. 1325–1328.
47. Marks L. ( 1974). "On associations of light and sound: the mediation of brightness, pitch, and loudness", American J. Psychol. Vol 87, 173188. http://dx.doi.org/10.2307/1422011
48. Marks L. ( 1987). "On cross-modal similarity: auditory–visual interactions in speeded discrimination", J. Exp. Psychol. Human Percept. Perform. Vol 13, 384394. http://dx.doi.org/10.1037/0096-1523.13.3.384
49. Martino G. , Marks L. ( 2000). "Cross-modal interaction between vision and touch: the role of synesthetic correspondence", Perception Vol 29, 745754. http://dx.doi.org/10.1068/p2984
50. Matteau I. , Kupers R. , Ricciardi E. , Pietrini P. , Ptito M. ( 2010). "Beyond visual, aural and haptic movement perception: hMT+ is activated by electrotactile motion stimulation of the tongue in sighted and in congenitally blind individuals", Brain Res. Bull. Vol 82, 264270. http://dx.doi.org/10.1016/j.brainresbull.2010.05.001
51. McMorrow G. , Wang X. , Whelan P. ( 1997). A color-to-speech sensory substitution device for the visually impaired, in: Machine Vision Applications, Architectures, and Systems Integration VI 1997, Pittsburgh, PA, USA, pp. 272–281.
52. Meers S. , Ward K. ( 2004). A vision system for providing 3D perception of the environment via transcutaneous electro-neural stimulation, in: Proceedings of the Information Visualisation, 8th International Conference, London, UK, pp. 546–552.
53. Meers S. , Ward K. ( 2005). A substitute vision system for providing 3d perception and gps navigation via electro-tactile stimulation, in: Proceedings of the International Conference on Sensing Technology, Palmerston North, New Zealand, pp. 551–556.
54. Meers S. , Ward K. ( 2007). Substitute three-dimensional perception using depth and colour sensors. Retrieved August 16th, 2012, from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1585&context=infopapers
55. Meijer P. ( 1992). "An experimental system for auditory image representations", IEEE Trans. Biomed. Eng. Vol 39, 112121. http://dx.doi.org/10.1109/10.121642
56. Melara R. , O’Brien T. ( 1987). "Interaction between synesthetically corresponding dimensions", J. Exp. Psychol. Gen. Vol 116, 323336. http://dx.doi.org/10.1037/0096-3445.116.4.323
57. Morgan G. , Goodson F. , Jones T. ( 1975). "Age differences in the associations between felt temperatures and color choices", Am. J. Psychol. Vol 88, 125130. http://dx.doi.org/10.2307/1421671
58. Morley J. , Rowe M. ( 1990). "Perceived pitch of vibrotactile stimuli — effects of vibration amplitude, and implications for vibration frequency coding", J. Physiol. — London Vol 431, 403416. http://dx.doi.org/10.1113/jphysiol.1990.sp018336
59. O’Regan K. , Noë A. ( 2001). "A sensorimotor account of vision and visual consciousness", Behav. Brain Sci. Vol 24, 9391031. http://dx.doi.org/10.1017/S0140525X01000115
60. Orwell J. , Remagnino P. M. , Jones G. A. ( 2001). "Optimal color quantization for real-time object recognition", Real-Time Imaging Vol 7, 401414. http://dx.doi.org/10.1006/rtim.2000.0209
61. Ou L. , Luo M. , Sun P. , Hu N. , Chen H. , Guan S. , Woodcock A. , Caivano J. , Huertas R. , Tremeau A. , Billger M. , Izaban H. , Richter K. ( 2010). "A cross-cultural comparison of colour emotion for two-colour combinations", Color Res. Appl. Vol 37, 2343. http://dx.doi.org/10.1002/col.20648
62. Regier T. , Kay P. , Cook R. ( 2005). "Focal colors are universal after all", Proc. Natl Acad. Sci. USA Vol 102, 83868391. http://dx.doi.org/10.1073/pnas.0503281102
63. Rivest J. , Cavanagh P. ( 1996). "Localizing contours defined by more than one attribute", Vis. Res. Vol 36, 5366. http://dx.doi.org/10.1016/0042-6989(95)00056-6
64. Roffler S. , Butler R. ( 1967). "Localization of tonal stimuli in the vertical plane", J. Acoust. Soc. Am. Vol 43, 12601266. http://dx.doi.org/10.1121/1.1910977
65. Rossi J. , Perales F. , Varona J. , Roca M. ( 2009). Col.diesis: transforming colour into melody and implementing the result in a colour sensor device, in: Proceedings for the 2nd International Conference in Visualisation, Barcelona, Spain, pp. 30–35.
66. Roth P. , Richoz D. , Petrucci L. , Pun T. ( 2001). An audio-haptic tool for non-visual image representation, in: International Symposium on Signal Processing and Its Applications (ISSPA), Kuala Lumpur, Malaysia, pp. 64–67.
67. Rousselet G. , Joubert O. , Fabre-Thorpe M. ( 2005). "How long to get to the “gist” of real-world natural scenes?" Vis. Cogn. Vol 12, 852877. http://dx.doi.org/10.1080/13506280444000553
68. Roy E. , Hollins M. ( 1998). "A ratio code for vibrotactile pitch", Somatosens. Mot. Res. Vol 15, 134145. http://dx.doi.org/10.1080/08990229870862
69. Sampaio E. , Maris S. , Bachy-y-Rita P. ( 2001). "Brain plasticity: “visual” acuity of blind persons via the tongue", Brain Res. Vol 908, 204207. http://dx.doi.org/10.1016/S0006-8993(01)02667-1
70. Schwerdt H. N. , Tapson J. , Etienne-Cummings R. ( 2009). A color detection glove with haptic feedback for the visually disabled, in: 43rd Annual Conference on Information Science and Systems, 2009, Baltimore, MD, USA, pp. 681–686.
71. Segond H. , Weiss D. , Sampaio E. ( 2005). "Human spatial navigation via a visuo-tactile sensory substitution system", Perception Vol 34, 12311249. http://dx.doi.org/10.1068/p3409
72. Shepard R. ( 1964). "Circularity in judgments of relative pitch", J. Acoust. Soc. Am. Vol 36, 23462353. http://dx.doi.org/10.1121/1.1919362
73. Sjöström C. , Rassmus-Gröhn K. ( 1999). "The sense of touch provides new computer interaction techniques for disabled people", Technol. Disabil. Vol 10, 4552.
74. Striem-Amit E. , Guendelman M. , Amedi A. ( 2012). "‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution", Plos ONE Vol 7, e33136. DOI: http://dx.doi.org/10.1371/journal.pone.0033136
75. Tapson J. , Gurari N. , Diaz J. , Chicca E. , Sander D. , Pouliquen P. , Etienne-Cummings R. ( 2008). The feeling of color: a haptic feedback device for the visually disabled, in: Proceedings for the IEEE Biomedical Circuits and Systems Conference, 2008. BioCAS, Seattle, WA, USA, pp. 281–384.
76. Taylor B. ( 1977). "Dimensional interactions in vibrotactile information-processing", Percept. Psychophys. Vol 21, 477481. http://dx.doi.org/10.3758/BF03199505
77. Torralba A. ( 2009). "How many pixels make an image?" Vis. Neurosci. Vol 26, 123131. http://dx.doi.org/10.1017/S0952523808080930
78. Vallbo A. , Johansson R. ( 1984). "Properties of cutaneous mechanoreceptors in the human hand related to touch sensation", Hum. Neurobiol. Vol 3, 314.
79. Verrillo R. , Fraioli A. , Smith R. ( 1969). "Sensation magnitude of vibrotactile stimuli", Percept. Psychophys. Vol 6, 366372. http://dx.doi.org/10.3758/BF03212793
80. Visell Y. ( 2009). "Tactile sensory substitution: models for enaction in HCI", Interact. Comput. Vol 21, 3853. http://dx.doi.org/10.1016/j.intcom.2008.08.004
81. Von Bismarck G. ( 1974). "Timbre of steady tones: a factorial investigation of its verbal attributes", Acustica Vol 30, 146159.
82. Walker P. , Bremner G. , Mason U. , Spring J. , Mattock K. , Slater A. , Johnson S. ( 2010). "Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences", Psychol. Sci. Vol 21, 2125. http://dx.doi.org/10.1177/0956797609354734
83. Wang D. , Brown G. ( 2006). Computational Auditory Scene Analysis: Principles, Algorithms, and Applications. Wiley-Blackwell, New York, NY, USA. http://dx.doi.org/10.1109/9780470043387
84. Ward J. , Wright T. (in press). "Sensory substitution as an artificially acquired synaesthesia", Neurosci. Biobehav. Rev.DOI:
85. Ward J. , Huckstep B. , Tsakanikos E. ( 2006). "Sound–colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all?" Cortex Vol 42, 264280. http://dx.doi.org/10.1016/S0010-9452(08)70352-6
86. Way T. P. , Barner K. E. ( 1997a). "Automatic visual to tactile translation. Part I: human factors, access methods, and image manipulation", IEEE Trans. Rehabil. Eng. Vol 5, 8194. http://dx.doi.org/10.1109/86.559353
87. Way T. P. , Barner K. E. ( 1997b). "Automatic visual to tactile translation. Part II: evaluation of the TACTile Image Creation System", IEEE Trans. Rehabil. Eng. Vol 5, 95105. http://dx.doi.org/10.1109/86.559354
88. Wessel D. ( 1985). "Timbre space as a musical control structure", in: Foundations of Computer Music, Roads C. , Strawn J. (Eds), pp.  4552. MIT Press, Cambridge, MA, USA.
89. WHO( 2012). Visual impairment and blindness. Retrieved August 16th, 2012, from http://www.who.int/mediacentre/factsheets/fs282/en/index.html
90. Yang G.-H. , Kyung K.-U. , Srinivasan M. A. , Kwon D.-S. ( 2006). Quantitative tactile display device with pin-array type tactile feedback and thermal feedback, in: 2006 IEEE International Conference on Robotics and Automation (ICRA), Vols 1–10, Orlando, FL, USA, pp. 3917–3922.
91. Zacharakis A. , Pastiadis K. , Papadelis G. , Reiss J. ( 2011). An investigation of musical timbre: uncovering salient semantic descriptors and perceptual dimensions, in: Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), Miami, FL, USA, pp. 807–812.
http://brill.metastore.ingenta.com/content/journals/10.1163/22134808-00002434
Loading

Article metrics loading...

/content/journals/10.1163/22134808-00002434
2013-01-01
2017-09-24

Sign-in

Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
     
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation