Cookies Policy

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

The Complex Interplay Between Multisensory Integration and Perceptual Awareness

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness.

Affiliations: 1: 1Centre for the Study of the Senses, Institute of Philosophy, School of Advanced Study, University of London, London, UK ; 2: 2Laboratory of Cognitive Neuroscience, Brain Mind Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland ; 3: 3Department of Translational Research on New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy ; 4: 4Crossmodal Research Laboratory, Department of Experimental Psychology, Oxford University, Oxford, UK ; 5: 5Computational Neuroscience and Cognitive Robotics Centre, University of Birmingham, Birmingham, UK

*Equal contributions, alphabetical order.
**To whom correspondence should be addressed. E-mail:,

Full text loading...


Data & Media loading...

1. Adam R. , Noppeney U. (2014). "A phonologically congruent sound boosts a visual target into perceptual awareness", Front. Integr. Neurosci. Vol 8, 70. DOI:.
2. Alais D. , Burr D. (2004). "Ventriloquist effect results from rear-optimal bimodal integration", Curr. Biol. Vol 14, 257262.
3. Allen J. , Kraus N. , Bradlow A. (2000). "Neural representation of consciously imperceptible speech sound differences", Percept. Psychophys. Vol 62, 13831393.
4. Aller M. , Giani A. , Conrad V. , Watanabe M. , Noppeney U. (2015). "A spatially collocated sound thrusts a flash into awareness", Front. Integr. Neurosci. Vol 9, 16. DOI:.
5. Alsius A. , Munhall K. G. (2013). "Detection of audiovisual speech correspondences without visual awareness", Psychol. Sci. Vol 24, 423431.
6. Andersen T. S. , Tiippana K. , Sams M. (2004). "Factors influencing audiovisual fission and fusion illusions", Cogn. Brain Res. Vol 21, 301308.
7. Arzi A. , Shedlesky L. , Ben-Shaul M. , Nasser K. , Oksenberg A. , Hairston I. S. , Sobel N. (2012). "Humans can learn new information during sleep", Nat. Neurosci. Vol 15, 14601465.
8. Astle D. E. , Nobre A. C. , Scerif G. (2010). "Subliminally presented and stored objects capture spatial attention", J. Neurosci. Vol 30, 35673571.
9. Balduzzi D. , Tononi G. (2008). "Integrated information in discrete dynamical systems: motivation and theoretical framework", PLoS Comput. Biol. Vol 4, e1000091. DOI:.
10. Bekinschtein T. A. , Dehaene S. , Rohaut B. , Tadel F. , Cohen L. , Naccache L. (2009). "Neural signature of the conscious processing of auditory regularities", Proc. Natl Acad. Sci. USA Vol 106, 16721677.
11. Bertelson P. , Aschersleben G. (1998). "Automatic visual bias of perceived auditory location", Psychonom. Bull. Rev. Vol 5, 482489.
12. Blake R. , Logothetis N. K. (2002). "Visual competition", Nat. Rev. Neurosci. Vol 3, 1321.
13. Blake R. , Sobel K. V. , James T. W. (2004). "Neural synergy between kinetic vision and touch", Psychol. Sci. Vol 15, 397402.
14. Blanke O. (2012). "Multisensory brain mechanisms of bodily self-consciousness", Nat. Rev. Neurosci. Vol 13, 556571.
15. Boring E. G. (1930). "A new ambiguous figure", Am. J. Psychol. Vol 42, 444445.
16. Bruno N. , Jacomuzzi A. , Bertamini M. , Meyer G. (2007). "A visual-haptic Necker cube reveals temporal constraints on intersensory merging during perceptual exploration", Neuropsychologia Vol 45, 469475.
17. Chalmers D. , Bayne T. (2003). "What is the unity of consciousness?", in: The Unity of Consciousness: Binding, Integration, and Dissociation, Cleeremans A. , Frith C. (Eds), pp.  2358. Oxford University Press, Oxford, UK.
18. Chen Y.-C. , Spence C. (2010). "When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked pictures", Cognition Vol 114, 389404.
19. Chen Y.-C. , Spence C. (2011a). "Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity", J. Exp. Psychol. Hum. Percept. Perform. Vol 37, 15541568.
20. Chen Y.-C. , Spence C. (2011b). "The crossmodal facilitation of visual object representations by sound: evidence from the backward masking paradigm", J. Exp. Psychol. Hum. Percept. Perform. Vol 37, 17841802.
21. Chen Y.-C. , Yeh S.-L. , Spence C. (2011). "Crossmodal constraints on human perceptual awareness: auditory semantic modulation of binocular rivalry", Front. Psychol. Vol 2, 212. DOI:.
22. Conrad V. , Bartels A. , Kleiner M. , Noppeney U. (2010). "Audiovisual interactions in binocular rivalry", J. Vis. Vol 10, 27. DOI:.
23. Conrad V. , Vitello M. P. , Noppeney U. (2012). "Interactions between apparent motion rivalry in vision and touch", Psychol. Sci. Vol 23, 940948.
24. Conrad V. , Kleiner M. , Bartels A. , Hartcher O’Brien J. , Bülthoff H. H. , Noppeney U. (2013). "Naturalistic stimulus structure determines the integration of audiovisual looming signals in binocular rivalry", PLoS ONE Vol 8, e70710. DOI:.
25. Critchley H. D. , Harrison N. A. (2013). "Visceral influences on brain and behavior", Neuron Vol 77, 624638.
26. Dayan P. (1998). "A hierarchical model of binocular rivalry", Neural Comput. Vol 10, 11191135.
27. De Graaf T. A. , Hsieh P. J. , Sack A. T. (2012). "The “correlates” in neural correlates of consciousness", Neurosci. Biobehav. Rev. Vol 36, 191197.
28. De Meo R. , Murray M. M. , Clarke S. , Matusz P. J. (2015). "Top-down control and early multisensory processes: chicken vs. egg", Front. Integr. Neurosci. Vol 9, 17. DOI:.
29. Dehaene S. (2001). "Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework", Cognition Vol 79, 137.
30. Dehaene S. , Changeux J.-P. (2011). "Experimental and theoretical approaches to conscious processing", Neuron Vol 70, 200227.
31. Deroy O. (2014). "The unity assumption and the many unities of consciousness", in: Sensory Integration and the Unity of Consciousness, Bennett D. , Hill C. (Eds), pp.  105124. MIT Press, Cambridge, MA, USA.
32. Deroy O. , Chen Y.-C. , Spence C. (2014). "Multisensory constraints on awareness", Phil. Trans. R. Soc. Lond. B, Biol. Sci. Vol 369(1641), 20130207. DOI:.
33. Di Luca M. , Ernst M. O. , Backus B. T. (2010). "Learning to use an invisible visual signal for perception", Curr. Biol. Vol 20, 18601863.
34. Di Pace E. , Saracini C. (2014). "Action imitation changes perceptual alternations in binocular rivalry", PLoS ONE Vol 9, e98305. DOI:.
35. Doner J. , Lappin J. S. , Perfetto G. (1984). "Detection of three-dimensional structure in moving optical patterns", J. Exp. Psychol. Hum. Percept. Perform. Vol 10, 111.
36. Ernst M. O. , Banks M. S. (2002). "Humans integrate visual and haptic information in a statistically optimal fashion", Nature Vol 415(6870), 429433.
37. Faivre N. , Mudrik L. , Schwartz N. , Koch C. (2014). "Multisensory integration in complete unawareness: evidence from audiovisual congruency priming", Psychol. Sci. Vol 25, 20062016.
38. Faivre N. , Salomon R. , Blanke O. (2015). "Visual consciousness and bodily self-consciousness", Curr. Opin. Neurobiol. Vol 28, 2328.
39. Gallace A. , Spence C. (2008). "The cognitive and neural correlates of “tactile consciousness”: a multisensory perspective", Conscious. Cogn. Vol 17, 370407.
40. Gallace A. , Spence C. (2014). In Touch With the Future. Oxford University Press, Oxford, UK.
41. Gau R. , Noppeney U. (2016). "How prior expectations shape multisensory perception", NeuroImage Vol 124, 876886.
42. Ghazanfar A. A. , Schroeder C. E. (2006). "Is neocortex essentially multisensory?" Trends Cogn. Sci. Vol 10, 278285.
43. Ghose G. M. , Maunsell J. (1999). "Specialized representations in visual cortex: a role for binding?" Neuron Vol 24, 7985.
44. Giani A. S. , Belardinelli P. , Ortiz E. , Kleiner M. , Noppeney U. (2015). "Detecting tones in complex auditory scenes", NeuroImage Vol 122, 203213.
45. Gómez C. , Argandoña E. D. , Solier R. G. , Angulo J. C. , Vázquez M. (1995). "Timing and competition in networks representing ambiguous figures", Brain Cogn. Vol 29, 103114.
46. Gutschalk A. , Micheyl C. , Oxenham A. J. (2008). "Neural correlates of auditory perceptual awareness under informational masking", PLoS Biol. Vol 6, 11561165.
47. Guzman-Martinez E. , Ortega L. , Grabowecky M. , Mossbridge J. , Suzuki S. (2012). "Interactive coding of visual spatial frequency and auditory amplitude-modulation rate", Curr. Biol. Vol 22, 383388.
48. Haynes J. D. , Driver J. , Rees G. (2005). "Visibility reflects dynamic changes of effective connectivity between V1 and fusiform cortex", Neuron Vol 46, 811821.
49. Heffner R. S. , Heffner H. E. (1992a). "Evolution of sound localization in mammals", in: The Evolutionary Biology of Hearing, Webster D. B. , Popper A. N. , Fay R. R. (Eds), pp.  691715. Springer Verlag, New York, NY, USA.
50. Heffner R. S. , Heffner H. E. (1992b). "Visual factors in sound localization in mammals", J. Comp. Neurol. Vol 317, 219232.
51. Hillis J. M. , Ernst M. O. , Banks M. S. , Landy M. S. (2002). "Combining sensory information: mandatory fusion within, but not between, senses", Science Vol 298(5598), 16271630.
52. Horlitz K. L. , O’Leary A. (1993). "Satiation or availability? Effects of attention, memory, and imagery on the perception of ambiguous figures", Percept. Psychophys. Vol 53, 668681.
53. Hsiao J. Y. , Chen Y.-C. , Spence C. , Yeh S. L. (2012). "Assessing the effects of audiovisual semantic congruency on the perception of a bistable figure", Conscious. Cogn. Vol 21, 775787.
54. Hsieh P.-J. , Colas J. T. (2012). "Awareness is necessary for extracting patterns in working memory but not for directing spatial attention", J. Exp. Psychol. Hum. Percept. Perform. Vol 38, 10851090.
55. Hsieh P.-J. , Colas J. T. , Kanwisher N. (2011). "Pop-out without awareness: unseen feature singletons capture attention only when top-down attention is available", Psychol. Sci. Vol 22, 12201226.
56. Jackson C. V. (1953). "Visual factors in auditory localization", Q. J. Exp. Psychol. Vol 5, 5265.
57. Kang M.-S. , Blake R. (2005). "Perceptual synergy between seeing and hearing revealed during binocular rivalry", Psychologija Vol 32, 715.
58. Kayser C. , Logothetis N. K. , Panzeri S. (2010). "Visual enhancement of the information representation in auditory cortex", Curr. Biol. Vol 20, 1924.
59. Knill D. C. , Pouget A. (2004). "The Bayesian brain: the role of uncertainty in neural coding and computation", Trends Neurosci. Vol 27, 712719.
60. Körding K. P. , Beierholm U. , Ma W. J. , Quartz S. , Tenenbaum J. B. , Shams L. (2007). "Causal inference in multisensory perception", PLoS ONE Vol 2, e943. DOI:.
61. Kruger H. , Collins T. , Cavanagh P. (2014). "Similar effects of saccades on auditory and visual localization suggest common spatial map", J. Vis. Vol 14, 1232.
62. Kuang S. , Zhang T. (2014). "Smelling directions: olfaction modulates ambiguous visual motion perception", Sci. Rep. Vol 4, 5796. DOI:.
63. Lamme V. A. F. (2006). "Towards a true neural stance on consciousness", Trends Cogn. Sci. Vol 10, 494501.
64. Lamme V. A. F. , Roelfsema P. R. (2000). "The distinct modes of vision offered by feedforward and recurrent processing", Trends Neurosci. Vol 23, 571579.
65. Lamy D. , Alon L. , Carmel T. , Shalev N. (2015). "The role of conscious perception in attentional capture and object-file updating", Psychol. Sci. Vol 26, 4857.
66. Lee H. , Noppeney U. (2011). "Long-term music training tunes how the brain temporally binds signals from multiple senses", Proc. Natl Acad. Sci. USA Vol 108, E1441E1450.
67. Lee H. , Noppeney U. (2014). "Temporal prediction errors in visual and auditory cortices", Curr. Biol. Vol 24, R309R310.
68. Lee M. , Blake R. , Kim S. , Kim C.-Y. (2015). "Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music", Proc. Natl Acad. Sci. USA Vol 112, 84938498.
69. Levelt W. J. (1965). On Binocular Rivalry. Institute for Perception, Soesterberg, Netherlands.
70. Liang M. , Mouraux A. , Hu L. , Iannetti G. D. (2013). "Primary sensory cortices contain distinguishable spatial patterns of activity for each sense", Nat. Commun. Vol 4, 1979. DOI:.
71. Liebert R. M. , Burk B. (1985). "Voluntary control of reversible figures", Percept. Mot. Skills Vol 61, 13071310.
72. Lunghi C. , Alais D. (2013). "Touch interacts with vision during binocular rivalry with a tight orientation tuning", PLoS ONE Vol 8, e58754. DOI:.
73. Lunghi C. , Alais D. (2015). "Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalry", Sci. Rep. Vol 5, 9413. DOI:.
74. Lunghi C. , Morrone M. C. (2013). "Early interaction between vision and touch during binocular rivalry", Multisens. Res. Vol 26, 291306.
75. Lunghi C. , Binda P. , Morrone M. C. (2010). "Touch disambiguates rivalrous perception at early stages of visual analysis", Curr. Biol. Vol 20, R143R144.
76. Lunghi C. , Morrone M. C. , Alais D. (2014). "Auditory and tactile signals combine to influence vision during binocular rivalry", J. Neurosci. Vol 34, 784792.
77. Maeda F. , Kanai R. , Shimojo S. (2004). "Changing pitch induced visual motion illusion", Curr. Biol. Vol 14, R990R991.
78. Maruya K. , Yang E. , Blake R. (2007). "Voluntary action influences visual competition", Psychol. Sci. Vol 18, 10901098.
79. McGurk H. , MacDonald J. (1976). "Hearing lips and seeing voices", Nature Vol 264(5588), 746748.
80. Meng M. , Tong F. (2004). "Can attention selectively bias bistable perception? Differences between binocular rivalry and ambiguous figures", J. Vis. Vol 4, 539551.
81. Meredith M. A. , Stein B. E. (1986). "Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration", J. Neurophysiol. Vol 56, 640662.
82. Meyer K. , Kaplan J. T. , Essex R. , Webber C. , Damasio H. , Damasio A. (2010). "Predicting visual stimuli on the basis of activity in auditory cortices", Nat. Neurosci. Vol 13, 667668.
83. Mudrik L. , Faivre N. , Koch C. (2014). "Information integration without awareness", Trends Cogn. Sci. Vol 18, 488496.
84. Munhall K. G. , Gribble P. , Sacco L. , Ward M. (1996). "Temporal constraints on the McGurk effect", Percept. Psychophys. Vol 58, 351362.
85. Munhall K. G. , ten Hove M. W. , Brammer M. , Paré M. (2009). "Audiovisual integration of speech in a bistable illusion", Curr. Biol. Vol 19, 735739.
86. Necker L. A. (1832). "Observations on some remarkable optical phænomena seen in Switzerland; and on an optical phænomenon which occurs on viewing a figure of a crystal or geometrical solid", Philos. Mag. Ser. 3 Vol 1, 329337.
87. Noel J.-P. , Wallace M. , Blake R. (2015). "Cognitive neuroscience: integration of sight and sound outside of awareness?" Curr. Biol. Vol 25, R157R159.
88. Olivers C. N. L. , Van der Burg E. (2008). "Bleeping you out of the blink: sound saves vision from oblivion", Brain Res. Vol 1242, 191199.
89. Paffen C. L. E. , Alais D. (2011). "Attentional modulation of binocular rivalry", Front. Hum. Neurosci. Vol 5, 105. DOI:.
90. Palmer T. D. , Ramsey A. K. (2012). "The function of consciousness in multisensory integration", Cognition Vol 125, 353364.
91. Partan S. , Marler P. (1999). "Communication goes multimodal", Science Vol 283(5406), 12721273.
92. Ro T. , Breitmeyer B. , Burton P. , Singhal N. S. , Lane D. (2003). "Feedback contributions to visual awareness in human occipital cortex", Curr. Biol. Vol 13, 10381041.
93. Rohe T. , Noppeney U. (2015). "Cortical hierarchies perform Bayesian causal inference in multisensory perception", PLoS Biol. Vol 13, e1002073. DOI:.
94. Rohe T. , Noppeney U. (2016). "Distinct computational principles govern multisensory integration in primary sensory and association cortices", Curr. Biol. Vol 1, 16.
95. Romei V. , Murray M. M. , Cappe C. , Thut G. (2009). "Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds", Curr. Biol. Vol 19, 17991805.
96. Roskies A. L. (1999). "The binding problem", Neuron Vol 24, 79.
97. Rubin E. (1915). Synsoplevede Figurer [Visually experienced Figures]. Studier i Psykologisk Analyse. Gyldendal, Copenhagen, Denmark.
98. Salomon R. , Lim M. , Herbelin B. , Hesselmann G. , Blanke O. (2013). "Posing for awareness: proprioception modulates access to visual consciousness in a continuous flash suppression task", J. Vis. Vol 13, 2.
99. Salomon R. , Kaliuzhna M. , Herbelin B. , Blanke O. (2015). "Balancing awareness: vestibular signals modulate visual consciousness in the absence of awareness", Conscious. Cogn. Vol 36, 289297.
100. Schroeder C. E. , Foxe J. (2005). "Multisensory contributions to low-level, “unisensory” processing", Curr. Opin. Neurobiol. Vol 15, 454458.
101. Shams L. , Kamitani Y. , Shimojo S. (2000). "Illusions. What you see is what you hear", Nature Vol 408(6814), 788.
102. Spence C. , Bayne T. (2015). "Is consciousness multisensory?", in: Perception and Its Modalities, Stokes D. , Matthen M. , Biggs S. (Eds), pp.  95132. Oxford University Press, Oxford, UK.
103. Spence C. , Smith B. , Auvray M. (2015). "Confusing tastes and flavours", in: Perception and Its Modalities, Stokes D. , Matthen M. , Biggs S. (Eds), pp.  247274. Oxford University Press, Oxford, UK.
104. Stanford T. R. (2005). "Evaluating the operations underlying multisensory integration in the cat superior colliculus", J. Neurosci. Vol 25, 64996508.
105. Stein B. E. , Meredith M. A. (1993). The Merging of the Senses. MIT Press, Cambridge, MA, USA.
106. Stein B. E. , Stanford T. R. (2008). "Multisensory integration: current issues from the perspective of the single neuron", Nat. Rev. Neurosci. Vol 9, 255266.
107. Stein B. E. , Stanford T. R. , Ramachandran R. , Perrault T. J. , Rowland B. A. (2009). "Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness", Exp. Brain Res. Vol 198, 113126.
108. Stevenson R. J. , Attuquayefio T. (2013). "Human olfactory consciousness and cognition: its unusual features may not result from unusual functions but from limited neocortical processing resources", Front. Psychol. Vol 4, 819. DOI:.
109. Tsuchiya N. , Koch C. (2005). "Continuous flash suppression reduces negative afterimages", Nat. Neurosci. Vol 8, 10961101.
110. Tsuchiya N. , Koch C. , Gilroy L. A. , Blake R. (2006). "Depth of interocular suppression associated with continuous flash suppression, flash suppression, and binocular rivalry", J. Vis. Vol 6, 10681078.
111. Van Ee R. , Van Boxtel J. J. , Parker A. L. , Alais D. (2009). "Multisensory congruency as a mechanism for attentional control over perceptual selection", J. Neurosci. Vol 29, 1164111649.
112. Vetter P. , Smith F. W. , Muckli L. (2014). "Decoding sound and imagery content in early visual cortex", Curr. Biol. Vol 24, 12561262.
113. Vidal M. , Barrès V. (2014). "Hearing (rivaling) lips and seeing voices: how audiovisual interactions modulate perceptual stabilization in binocular rivalry", Front. Hum. Neurosci. Vol 8, 677. DOI:.
114. von Saldern S. , Noppeney U. (2013). "Sensory and striatal areas integrate auditory and visual signals into behavioral benefits during motion discrimination", J. Neurosci. Vol 33, 88418849.
115. Vroomen J. , De Gelder B. (2004). "Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon", in: The Handbook of Multisensory Processes, Calvert G. A. , Spence C. , Stein B. E. (Eds), pp.  141146. MIT Press, Cambridge, MA, USA.
116. Werner S. , Noppeney U. (2010a). "Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization", J. Neurosci. Vol 30, 26622675.
117. Werner S. , Noppeney U. (2010b). "Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization", Cereb. Cortex Vol 20, 18291842.
118. Wolfe J. M. , Cave K. R. (1999). "The psychophysical evidence for a binding problem in human vision", Neuron Vol 24, 1117.
119. Yang Y. H. , Yeh S. L. (2014). "Unmasking the dichoptic mask by sound: spatial congruency matters", Exp. Brain Res. Vol 232, 11091116.
120. Zhang X. , Fang F. (2012). "Object-based attention guided by an invisible object", Exp. Brain Res. Vol 223, 397404.
121. Zhou W. , Jiang Y. , He S. , Chen D. (2010). "Olfaction modulates visual perception in binocular rivalry", Curr. Biol. Vol 20, 13561358.

Article metrics loading...



Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation