Cookies Policy

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Statistically Optimal Multisensory Cue Integration: A Practical Tutorial

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Humans combine redundant multisensory estimates into a coherent multimodal percept. Experiments in cue integration have shown for many modality pairs and perceptual tasks that multisensory information is fused in a statistically optimal manner: observers take the unimodal sensory reliability into consideration when performing perceptual judgments. They combine the senses according to the rules of Maximum Likelihood Estimation to maximize overall perceptual precision. This tutorial explains in an accessible manner how to design optimal cue integration experiments and how to analyse the results from these experiments to test whether humans follow the predictions of the optimal cue integration model. The tutorial is meant for novices in multisensory integration and requires very little training in formal models and psychophysical methods. For each step in the experimental design and analysis, rules of thumb and practical examples are provided. We also publish Matlab code for an example experiment on cue integration and a Matlab toolbox for data analysis that accompanies the tutorial online. This way, readers can learn about the techniques by trying them out themselves. We hope to provide readers with the tools necessary to design their own experiments on optimal cue integration and enable them to take part in explaining when, why and how humans combine multisensory information optimally.


Full text loading...


Data & Media loading...

1. Ackermann J. F. , Landy M. S. (2013). "Choice of saccade endpoint under risk", J. Vis. Vol 13, 27. DOI:.
2. Adams W. J. , Graf E. W. , Ernst M. O. (2004). "Experience can change the “light-from-above” prior", Nat. Neurosci. Vol 7, 10571058.
3. Alais D. , Burr D. (2004). "The ventriloquist effect results from near-optimal bimodal integration", Curr. Biol. Vol 14, 257262.
4. Bresciani J.-P. , Ernst M. O. , Drewing K. , Bouyer G. , Maury V. , Kheddar A. (2005). "Feeling what you hear: auditory signals can modulate tactile tap perception", Exp. Brain Res. Vol 162, 172180.
5. Burge J. , Ernst M. O. , Banks M. S. (2008). "The statistical determinants of adaptation rate in human reaching", J. Vis. Vol 8, 20, 119. DOI:.
6. Burge J. , Girshick A. R. , Banks M. S. (2010). "Visual–haptic adaptation is determined by relative reliability", J. Neurosci. Vol 30, 77147721.
7. Clark J. J. , Yuille A. L. (1990). Data Fusion for Sensory Information Processing Systems. Kluwer Academic, Boston, MA, USA.
8. Diedrichsen J. (2007). "Optimal task-dependent changes of bimanual feedback control and adaptation", Curr. Biol. Vol 17, 16751679.
9. Ernst M. O. (2006). "A Bayesian view on multimodal cue integration", in: Human Body Perception from the Inside Out, Knoblich G. (Ed.), pp.  105131. Oxford University Press, New York, NY, USA.
10. Ernst M. O. (2007). "Learning to integrate arbitrary signals from vision and touch", J. Vis. Vol 7, 7, 114. DOI:.
11. Ernst M. O. (2012). "Optimal multisensory integration: assumptions and limits", in: The New Handbook of Multisensory Processes, Stein B. E. (Ed.), pp.  10841124. MIT Press, Cambridge, MA, USA.
12. Ernst M. O. , Banks M. S. (2002). "Humans integrate visual and haptic information in a statistically optimal fashion", Nature Vol 415(6870), 429433.
13. Ernst M. O. , Bülthoff H. H. (2004). "Merging the senses into a robust percept", Trends Cogn. Sci. Vol 8, 162169.
14. Fetsch C. R. , DeAngelis G. C. , Angelaki D. E. (2010). "Visual–vestibular cue integration for heading perception: applications of optimal cue integration theory", Eur. J. Neurosci. Vol 31, 17211729.
15. Gepshtein S. , Burge J. , Ernst M. O. , Banks M. S. (2005). "The combination of vision and touch depends on spatial proximity", J. Vis. Vol 5, 7, 10131023. DOI:.
16. Gescheider G. (1997). Psychophysics: the Fundamentals, 3rd edn. Lawrence Erlbaum Associates, Mahwah, NJ, USA.
17. Hartcher-O’Brien J. , Di Luca M. , Ernst M. O. (2014). "The duration of uncertain times: audiovisual information about intervals is integrated in a statistically optimal fashion", PLoS One Vol 9, e89339. DOI:.
18. Helbig H. B. , Ernst M. O. (2007). "Knowledge about a common source can promote visual–haptic integration", Perception Vol 36, 15231533.
19. Hillis J. M. , Ernst M. O. , Banks M. S. , Landy M. S. (2002). "Combining sensory information: mandatory fusion within, but not between, senses", Science Vol 298(5598), 16271630.
20. Hillis J. M. , Watt S. J. , Landy M. S. , Banks M. S. (2004). "Slant from texture and disparity cues: optimal cue combination", J. Vis. Vol 4, 967992.
21. Kleiner M. , Brainard D. , Pelli D. , Ingling A. , Murray R. , Broussard C. (2007). "What’s new in Psychtoolbox-3?" Perception Vol 36, ECVP Abstract Supplement.
22. Knill D. C. (1998). "Discrimination of planar surface slant from texture: human and ideal observers compared", Vis. Res. Vol 38, 16831711.
23. Knill D. C. , Saunders J. A. (2003). "Do humans optimally integrate stereo and texture information for judgments of surface slant?" Vis. Res. Vol 43, 5392558.
24. Körding K. P. , Beierholm U. , Ma W. J. , Quartz S. , Tenenbaum J. B. , Shams L. (2007). "Causal inference in multisensory perception", PLoS One Vol 2, e943. DOI:.
25. Landy M. S. , Maloney L. T. , Johnston E. B. , Young M. J. (1995). "Measurement and modeling of depth cue combination: in defense of weak fusion", Vis. Res. Vol 35, 389412.
26. Landy M. S. , Banks M. S. , Knill D. C. (2011). "Ideal-observer models of cue integration", in: Sensory Cue Integration, Trommershäuser J. , Körding K. , Landy M. S. (Eds), pp.  529. Oxford University Press, New York, NY, USA.
27. Moscatelli A. , Mezzetti M. , Lacquaniti F. (2012). "Modeling psychophysical data at the population-level: the generalized linear mixed model", J. Vis. Vol 12, 26. DOI:.
28. Najemnik J. , Geisler W. S. (2005). "Optimal eye movement strategies in visual search", Nature Vol 434(7031), 387391.
29. Najemnik J. , Geisler W. S. (2009). "Simple summation rule for optimal fixation selection in visual search", Vis. Res. Vol 49, 12861294.
30. Newell F. N. , Ernst M. O. , Tjan B. S. , Bülthoff H. H. (2001). "Viewpoint dependence in visual and haptic object recognition", Psychol. Sci. Vol 12, 3742.
31. Oruç I. , Maloney L. T. , Landy M. S. (2003). "Weighted linear cue combination with possibly correlated error", Vis. Res. Vol 43, 24512468.
32. Parise C. V. , Spence C. , Ernst M. O. (2012). "When correlation implies causation in multisensory integration", Curr. Biol. Vol 22, 4649.
33. Parise C. V. , Knorre K. , Ernst M. O. (2014). "Natural auditory scene statistics shapes human spatial hearing", Proc. Natl Acad. Sci. USA Vol 111, 61046108.
34. Plaisier M. A. , Van Dam L. C. J. , Glowania C. , Ernst M. O. (2014). "Exploration mode affects visuohaptic integration of surface orientation", J. Vis. Vol 14, 22, 112. DOI:.
35. Roach N. W. , Heron J. , McGraw P. V. (2006). "Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration", Proc. Biol. Sci. Vol 273(1598), 21592168.
36. Rock I. , Victor J. (1964). "Vision and touch: an experimentally created conflict between the two senses", Science Vol 143(3606), 594596.
37. Shams L. , Kamitani Y. , Shimojo S. (2002). "Visual illusion induced by sound", Cogn. Brain Res. Vol 14, 147152.
38. Todorov E. , Jordan M. I. (2002). "Optimal feedback control as a theory of motor coordination", Nat. Neurosci. Vol 5, 12261235.
39. Trommershäuser J. , Maloney L. T. , Landy M. S. (2003). "Statistical decision theory and the selection of rapid, goal-directed movements", J. Opt. Soc. Am. A Vol 20, 14191433.
40. Van Beers R. J. , Sittig A. C. , Van der Gon J. J. D. (1999). "Integration of proprioceptive and visual position-information: an experimentally supported model", J. Neurophysiol. Vol 81, 13551364.
41. Van Dam L. C. J. , Ernst M. O. (2013). "Knowing each random error of our ways, but hardly correcting for it: an instance of optimal performance", PLoS One Vol 8, e78757. DOI:.
42. Van Dam L. C. J. , Rohde M. (2015). Maximum Likelihood Multisensory Integration Toolbox () Matlab Central File Exchange, retrieved 18.4.2015.
43. Van Dam L. C. J. , Parise C. V. , Ernst M. O. (2014). "Modeling multisensory integration", in: Sensory Integration and the Unity of Consciousness, Bennett D. J. , Hill C. S. (Eds), pp.  209229. MIT Press, Cambridge MA, USA.
44. Weiss Y. , Simoncelli E. P. , Adelson E. H. (2002). "Motion illusions as optimal percepts", Nat. Neurosci. Vol 5, 598604.
45. Wichmann F. A. , Hill N. J. (2001). "The psychometric function: I. Fitting, sampling and goodness of fit", Percept. Psychophys. Vol 63, 12931313.
46. Yuille A. L. , Bülthoff H. H. (1996). "Bayesian decision theory and psychophysics", in: Bayesian Perspectives on Visual Perception, Knill D. C. , Richards W. (Eds), pp.  123161. Cambridge University Press, Cambridge, MA, USA.

Article metrics loading...



Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation