Cookies Policy

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Electrophysiological Indices of Audiovisual Speech Perception: Beyond the McGurk Effect and Speech in Noise

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Access this article

+ Tax (if applicable)
Add to Favorites
You must be logged in to use this functionality

image of Multisensory Research
For more content, see Seeing and Perceiving and Spatial Vision.

Visual information on a talker’s face can influence what a listener hears. Commonly used approaches to study this include mismatched audiovisual stimuli (e.g., McGurk type stimuli) or visual speech in auditory noise. In this paper we discuss potential limitations of these approaches and introduce a novel visual phonemic restoration method. This method always presents the same visual stimulus (e.g., /ba/) dubbed with a matched auditory stimulus (/ba/) or one that has weakened consonantal information and sounds more /a/-like). When this reduced auditory stimulus (or /a/) is dubbed with the visual /ba/, a visual influence will result in effectively ‘restoring’ the weakened auditory cues so that the stimulus is perceived as a /ba/. An oddball design in which participants are asked to detect the /a/ among a stream of more frequently occurring /ba/s while either a speaking face or face with no visual speech was used. In addition, the same paradigm was presented for a second contrast in which participants detected /pa/ among /ba/s, a contrast which should be unaltered by the presence of visual speech. Behavioral and some ERP findings reflect the expected phonemic restoration for the /ba/ vs. /a/ contrast; specifically, we observed reduced accuracy and P300 response in the presence of visual speech. Further, we report an unexpected finding of reduced accuracy and P300 response for both speech contrasts in the presence of visual speech, suggesting overall modulation of the auditory signal in the presence of visual speech. Consistent with this, we observed a mismatch negativity (MMN) effect for the /ba/ vs. /pa/ contrast only that was larger in absence of visual speech. We discuss the potential utility for this paradigm for listeners who cannot respond actively, such as infants and individuals with developmental disabilities.

Affiliations: 1: 1Haskins Laboratories, New Haven, CT, USA ; 2: 3Southern Connecticut State University, New Haven, CT, USA ; 3: 2University of Connecticut, Storrs, CT, USA

*To whom correspondence should be addressed. E-mail:
Loading data from figshare Loading data from figshare

Full text loading...


Data & Media loading...

1. Alcántara J. I., Weisblatt E. J., Moore B. C., Bolton P. F. (2004). "Speech-in-noise perception in high-functioning individuals with autism or Asperger’s syndrome", J. Child Psychol. Psychiat. Vol 45, 11071114. [Crossref]
2. Alho K. (1995). "Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes", Ear Hear. Vol 16, 3851. [Crossref]
3. Alsius A., Navarra J., Campbell R., Soto-Faraco S. (2005). "Audiovisual integration of speech falters under high attention demands", Curr. Biol. Vol 15, 839843. [Crossref]
4. Bebko J. M., Weiss J. A., Demark J. L., Gomez P. (2006). "Discrimination of temporal synchrony in intermodal events by children with autism and children with developmental disabilities without autism", J. Child Psychol. Psychiat. Vol 47, 8898. [Crossref]
5. Bergeson T. R., Pisoni D. B. (2004). "Audiovisual speech perception in deaf adults and children following cochlear implantation", in: Handbook of Multisensory Processes, Calvert G., Sence C., Stein B. E. (Eds), pp.  749771. MIT Press, Cambridge, MA, USA.
6. Bernstein L. E., Ponton C. W., Auer E. T. Jr (2001). Electrophysiology of unimodal and audiovisual speech perception, in: AVSP 2001 — International Conference on Auditory–Visual Speech Processing, Aalborg, Denmark, pp. 50–55.
7. Bernstein L. E., Auer E. T., Wagner M., Ponton C. W. (2008). "Spatiotemporal dynamics of audiovisual speech processing", Neuroimage Vol 39, 423435. [Crossref]
8. Boersma P., Weenink D. (2013). Praat: doing phonetics by computer, version 5.3.39, University of Amsterdam, Amsterdam, Netherlands. Retrieved from, September 26, 2016.
9. Brancazio L. (2004). "Lexical influences in audiovisual speech perception", J. Exp. Psychol. Hum. Percept. Perform. Vol 30, 445463. [Crossref]
10. Brancazio L., Miller J. L. (2005). "Use of visual information in speech perception: evidence for a visual rate effect both with and without a McGurk effect", Percept. Psychophys. Vol 67, 759769. [Crossref]
11. Brancazio L., Best C. T., Fowler C. A. (2006). "Visual influences on perception of speech and nonspeech vocal-tract events", Lang Speech Vol 49, 2153. [Crossref]
12. Brancazio L., Moore D., Tyska K., Burke S., Cosgrove D., Irwin J. (2015). McGurk-like effects of subtle audiovisual mismatch in speech perception, presented at the 27th Annual Convention of the Association for Psychological Science, New York, NY, USA, May 23, 2015.
13. Colin C., Radeau M., Soquet A., Demolin D., Colin F., Deltenre P. (2002). "Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory", Clin. Neurophysiol. Vol 113, 495506. [Crossref]
14. Desjardins R. N., Rogers J., Werker J. F. (1997). "An exploration of why preschoolers perform differently than do adults in audiovisual speech perception tasks", J. Exp. Child Psychol. Vol 66, 85110. [Crossref]
15. Eigsti I. M., Shapiro T. (2003). "A systems neuroscience approach to autism: biological, cognitive, and clinical perspectives", Ment. Retard. Dev. Disabil. Res. Rev. Vol 9, 205215. [Crossref]
16. Erber N. P. (1975). "Auditory–visual perception of speech", J. Speech Hear. Disord. Vol 40, 481492. [Crossref]
17. Ferree T. C., Luu P., Russell G. S., Tucker D. M. (2001). "Scalp electrode impedance, infection risk, and EEG data quality", Clin. Neurophysiol. Vol 112, 536544. [Crossref]
18. Foss-Feig J. H., Kwakye L. D., Cascio C. J., Burnette C. P., Kadivar H., Stone W. L., Wallace M. T. (2010). "An extended multisensory temporal binding window in autism spectrum disorders", Exp. Brain Res. Vol 203, 381389. [Crossref]
19. Grant K. W., Seitz P. F. (2000). "The use of visible speech cues for improving auditory detection of spoken sentences", J. Acoust. Soc. Am. Vol 108, 11971208. [Crossref]
20. Green K. (1994). "The influence of an inverted face on the McGurk effect", J. Acoust. Soc. Am. Vol 95, 3014. DOI:10.1121/1.408802. [Crossref]
21. Iarocci G., Rombough A., Yager J., Weeks D. J., Chua R. (2010). "Visual influences on speech perception in children with autism", Autism Vol 14, 305320. [Crossref]
22. Irwin J. R., Tornatore L. A., Brancazio L., Whalen D. H. (2011). "Can children with autism spectrum disorders “hear” a speaking face?" Child Dev. Vol 82, 13971403. [Crossref]
23. Jerger S., Damian M. F., Tye-Murray N., Abdi H. (2014). "Children use visual speech to compensate for non-intact auditory speech", J. Exp. Child Psychol. Vol 126, 295312. [Crossref]
24. Kaganovich N., Schumaker J., Leonard L. B., Gustafson D., Macias D. (2014). "Children with a history of SLI show reduced sensitivity to audiovisual temporal asynchrony: an ERP study", J. Speech Lang. Hear. Res. Vol 57, 14801502. [Crossref]
25. Kaganovich N., Schumaker J., Rowland C. (2016). "Matching heard and seen speech: an ERP study of audiovisual word recognition", Brain Lang. Vol 157, 1424. [Crossref]
26. Kashino M. (2006). "Phonemic restoration: the brain creates missing speech sounds", Acoust. Sci. Technol. Vol 27, 318321. [Crossref]
27. Klucharev V., Möttönen R., Sams M. (2003). "Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception", Cogn. Brain Res. Vol 18, 6575. [Crossref]
28. Lachs L., Pisoni D. B., Kirk K. I. (2001). "Use of audiovisual information in speech perception by prelingually deaf children with cochlear implants: a first report", Ear Hear. Vol 22, 236251. [Crossref]
29. Legerstee M. (1990). "Infants use multimodal information to imitate speech sounds", Infant Behav. Dev. Vol 13, 343354. [Crossref]
30. Lewkowicz D. J., Hansen-Tift A. M. (2012). "Infants deploy selective attention to the mouth of a talking face when learning speech", Proc. Natl Acad. Sci. USA Vol 109, 14311436. [Crossref]
31. MacDonald J., McGurk H. (1978). "Visual influences on speech perception processes", Atten. Percept. Psychophys. Vol 24, 253257. [Crossref]
32. MacLeod A., Summerfield Q. (1987). "Quantifying the contribution of vision to speech perception in noise", Br. J. Audiol. Vol 21, 131141. [Crossref]
33. McGurk H., MacDonald J. (1976). "Hearing lips and seeing voices", Nature Vol 264(5588), 746748. [Crossref]
34. Meltzoff A. N., Kuhl P. K. (1994). "Faces and speech: intermodal processing of biologically relevant signals in infants and adults", in: The Development of Intersensory Perception: Comparative Perspectives, Lewkowicz D. J., Lickliter R. (Eds), pp.  335369. Erlbaum, Hillsdale, NJ, USA.
35. Ménard L., Dupont S., Baum S. R., Aubin J. (2009). "Production and perception of French vowels by congenitally blind adults and sighted adults", J. Acoust. Soc. Am. Vol 126, 14061414. [Crossref]
36. Molholm S., Ritter W., Murray M. M., Javitt D. C., Schroeder C. E., Foxe J. J. (2002). "Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study", Cogn. Brain Res. Vol 14, 115128. [Crossref]
37. Nath A. R., Beauchamp M. S. (2012). "A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion", Neuroimage Vol 59, 781787. [Crossref]
38. Payton K. L., Uchanski R. M., Braida L. D. (1994). "Intelligibility of conversational and clear speech in noise and reverberation for listeners with normal and impaired hearing", J. Acoust. Soc. Am. Vol 95, 15811592. [Crossref]
39. Pilling M. (2009). "Auditory event-related potentials (ERPs) in audiovisual speech perception", J. Speech Lang. Hear. Res. Vol 52, 10731081. [Crossref]
40. Pizzagalli D. A. (2007). "Electroencephalography and high-density electrophysiological source localization", in: Handbook of Psychophysiology, 3rd edn., Cacioppo J. T., Tassinary L. G., Berntson G. G. (Eds), pp.  5684. Cambridge University Press, Cambridge, UK. [Crossref]
41. Polich J. (2007). "Updating P300: an integrative theory of P3a and P3b", Clin. Neurophysiol. Vol 118, 21282148. [Crossref]
42. Rosenblum L. D. (2008). "Speech perception as a multimodal phenomenon", Curr. Dir. Psychol. Sci. Vol 17, 405409. [Crossref]
43. Ross L. A., Saint-Amour D., Leavitt V. M., Javitt D. C., Foxe J. J. (2007). "Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments", Cereb. Cortex Vol 17, 11471153. [Crossref]
44. Saint-Amour D., De Sanctis P., Molholm S., Ritter W., Foxe J. J. (2007). "Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion", Neuropsychologia Vol 45, 587597. [Crossref]
45. Samuel A. G. (1981). "The role of bottom-up confirmation in the phonemic restoration illusion", J. Exp. Psychol. Hum. Percept. Perform. Vol 7, 11241131. [Crossref]
46. Schwartz J. L. (2010). "A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent", J. Acoust. Soc. Am. Vol 127, 15841594. [Crossref]
47. Smith E. G., Bennetto L. (2007). "Audiovisual speech integration and lipreading in autism", J. Child Psychol. Psychiat. Vol 48, 813821. [Crossref]
48. Soto-Faraco S., Alsius A. (2009). "Deconstructing the McGurk–MacDonald illusion", J. Exp. Psychol. Hum. Percept. Perform. Vol 35, 580587. [Crossref]
49. Sumby W. H., Pollack I. (1954). "Visual contribution to speech intelligibility in noise", J. Acoust. Soc. Am. Vol 26, 212215. [Crossref]
50. Tremblay K., Kraus N., McGee T., Ponton C., Otis B. (2001). "Central auditory plasticity: changes in the N1–P2 complex after speech-sound training", Ear Hear. Vol 22, 7990. [Crossref]
51. Van Wassenhove V., Grant K. W., Poeppel D. (2005). "Visual speech speeds up the neural processing of auditory speech", Proc. Natl Acad. Sci. USA Vol 102, 11811186. [Crossref]
52. Walker S., Bruce V., O’Malley C. (1995). "Facial identity and facial speech processing: familiar faces and voices in the McGurk effect", Percept. Psychophys. Vol 57, 11241133. [Crossref]
53. Warren R. M. (1970). "Perceptual restoration of missing speech sounds", Science Vol 167, 392393. [Crossref]
54. Windmann S. (2004). "Effects of sentence context and expectation on the McGurk illusion", J. Mem. Lang. Vol 50, 212230. [Crossref]

Article metrics loading...



Can't access your account?
  • Tools

  • Add to Favorites
  • Printable version
  • Email this page
  • Subscribe to ToC alert
  • Get permissions
  • Recommend to your library

    You must fill out fields marked with: *

    Librarian details
    Your details
    Why are you recommending this title?
    Select reason:
    Multisensory Research — Recommend this title to your library
  • Export citations
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation