Cookies Policy
X

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies.

I accept this policy

Find out more here

Open Access Language–General Auditory–Visual Speech Perception: Thai–English and Japanese–English McGurk Effects

No metrics data to plot.
The attempt to load metrics for this article has failed.
The attempt to plot a graph for these metrics has failed.
The full text of this article is not currently available.

Brill’s MyBook program is exclusively available on BrillOnline Books and Journals. Students and scholars affiliated with an institution that has purchased a Brill E-Book on the BrillOnline platform automatically have access to the MyBook option for the title(s) acquired by the Library. Brill MyBook is a print-on-demand paperback copy which is sold at a favorably uniform low price.

Language–General Auditory–Visual Speech Perception: Thai–English and Japanese–English McGurk Effects

  • PDF
  • HTML
Add to Favorites
You must be logged in to use this functionality

Affiliations: 1: 1MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, Australia ; 2: 2University of Queensland, Brisbane, Australia

*To whom correspondence should be addressed. E-mail: denis.burnham@westernsydney.edu.au

Cross-language McGurk Effects are used to investigate the locus of auditory–visual speech integration. Experiment 1 uses the fact that [], as in ‘sing’, is phonotactically legal in word-final position in English and Thai, but in word-initial position only in Thai. English and Thai language participants were tested for ‘n’ perception from auditory [m]/visual [] (A[m]V[]) in word-initial and -final positions. Despite English speakers’ native language bias to label word-initial [] as ‘n’, the incidence of ‘n’ percepts to A[m]V[] was equivalent for English and Thai speakers in final and initial positions. Experiment 2 used the facts that (i) [ð] as in ‘that’ is not present in Japanese, and (ii) English speakers respond more often with ‘tha’ than ‘da’ to A[ba]V[ga], but more often with ‘di’ than ‘thi’ to A[bi]V[gi]. English and three groups of Japanese language participants (Beginner, Intermediate, Advanced English knowledge) were presented with A[ba]V[ga] and A[bi]V[gi] by an English (Experiment 2a) or a Japanese (Experiment 2b) speaker. Despite Japanese participants’ native language bias to perceive ‘d’ more often than ‘th’, the four groups showed a similar phonetic level effect of [a]/[i] vowel context × ‘th’ vs. ‘d’ responses to A[b]V[g] presentations. In Experiment 2b this phonetic level interaction held, but was more one-sided as very few ‘th’ responses were evident, even in Australian English participants. Results are discussed in terms of a phonetic plus postcategorical model, in which incoming auditory and visual information is integrated at a phonetic level, after which there are post-categorical phonemic influences.

Loading

Full text loading...

/deliver/journals/22134808/22134808_00002590_text.html?itemId=/content/journals/10.1163/22134808-00002590&mimeType=html&fmt=ahah
/content/journals/10.1163/22134808-00002590
Loading

Data & Media loading...

http://brill.metastore.ingenta.com/content/journals/10.1163/22134808-00002590
Loading
Loading

Article metrics loading...

/content/journals/10.1163/22134808-00002590
2017-10-23
2017-11-22
1. Alsius A., Paré M., Munhall K. G. (2017). "Forty years after Hearing lips and seeing voices: the McGurk effect revisited", Multisens. Res. DOI:10.1163/22134808-00002565.
2. Baart M., Stekelenburg J. J., Vroomen J. (2014). "Electrophysiological evidence for speech specific audiovisual integration", Neuropsychologia Vol 53, 115121. [Crossref]
3. Bernstein L. E., Burnham D., Schwartz J.-L. (2002). Special session: Issues in audiovisual spoken language processing (When, Where, and How?), in: 7th International Conference on Spoken Language Processing, Denver, CO, USA. pp. 1445–1448. ISBN: 1876346-40-X.
4. Burnham D. (1998). "Language specificity in the development of auditory–visual speech perception", in: Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory–Visual Speech, Campbell R., Dodd B., Burnham D. (Eds), pp.  2760. Psychology Press, Hove, UK.
5. Burnham D., Dodd B. (1996). "Auditory–visual speech perception as a direct process: the McGurk effect in human infants and across languages", in: Speechreading by Humans and Machines, Stork D. G., Hennecke M. E. (Eds), pp.  103114. Springer-Verlag, Berlin, Germany. [Crossref]
6. Burnham D., Dodd B. (1998). "Familiarity and novelty in infant cross-language studies: factors, problems, and a possible solution", in: Advances in Infancy Research, Vol Vol. 12, Rovee-Collier C., Lipsitt L., Hayne H. (Eds), pp.  170187. Ablex Publishing Corporation, Stanford, CT, USA.
7. Burnham D., Dodd B. (2004). "Auditory–visual speech integration by pre-linguistic infants: perception of an emergent consonant in the McGurk effect", Dev. Psychobiol. Vol 44, 204220. [Crossref]
8. Burnham D., Keane S. (1997). The Japanese McGurk effect: the role of linguistic and cultural factors in auditory–visual speech perception, in: Proceedings of the Workshop on Auditory–Visual Speech Processing: Cognitive and Computational Approaches, Rhodes, Greece, pp. 93–96.
9. Burnham D., Tyler M., Horlyck S. (2002). "Periods of speech perception development and their vestiges in adulthood", in: An Integrated View of Language Development: Papers in Honor of Henning Wode, Burmeister P., Piske T., Rohde A. (Eds), pp.  281300. Wissenschaftlicher Verlag Trier, Trier, Germany.
10. Calvert G. A., Bullmore E. T., Brammer M. J., Campbell R., Williams S. C. R., McGuire P. K., Woodruff P. W. R., Iversen S. D., David A. S. (1997). "Activation of auditory cortex during silent lipreading", Science Vol 276, 593596. [Crossref]
11. Campbell R., MacSweeney M., Surguladzec S., Calvert G., McGuire P., Suckling J., Brammer M. J., David A. S. (2001). "Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning)", Cogn. Brain Res. Vol 12, 233243. [Crossref]
12. Choi J., Cutler A., Broersma M. (2017). "Early development of abstract language knowledge: evidence from perception–production transfer of birth-language memory", R. Soc. Open Sci. Vol 4, 160660. DOI:10.1098/rsos.160660.
13. Day R. H., Dickenson R. G. (1976). "The components of the Poggendorff illusion", Br. J. Psychol. Vol 67, 537552. [Crossref]
14. Desjardins R. N., Werker J. F. (2004). "Is the integration of heard and seen speech mandatory for infants?" Dev. Psychobiol. Vol 45, 187203. [Crossref]
15. Dodd B., Burnham D. (1988). "Processing speechread information", Volta Rev. N. Reflect. Speechread. Vol 90, 4560.
16. Green K. (1996). "The use of auditory and visual information in phonetic perception", in: Speechreading by Humans and Machines, Stork D. G., Hennecke M. E. (Eds), pp.  5577. Springer-Verlag, Berlin, Germany. [Crossref]
17. Green K. (1998). "The use of auditory and visual information during phonetic processing: implications for theories of speech perception", in: Hearing by Eye II: Advances in the Psychology of Speechreading and Auditory–Visual Speech, Campbell R., Dodd B., Burnham D. (Eds), pp.  325. Psychology Press, Hove, UK.
18. Green K. P., Kuhl P. K. (1989). "The role of visual information in the processing of place and manner features in speech perception", Percept. Psychophys. Vol 45, 3442. [Crossref]
19. Green K. P., Kuhl P. K. (1991). "Integral processing of visual place and auditory voicing information during phonetic perception", J. Exp. Psychol. Hum. Percept. Perform. Vol 17, 278288. [Crossref]
20. Green K. P., Miller J. L. (1985). "On the role of visual rate information in phonetic perception", Percept. Psychophys. Vol 38, 269276. [Crossref]
21. Green K. P., Norrix L. W. (1997). "Acoustic clues to place of articulation and the McGurk effect: the role of release bursts, aspiration, and formant transitions", J. Speech Lang. Hear. Res. Vol 40, 646665. [Crossref]
22. Green K. P., Kuhl P. K., Meltzoff A. N., Stevens E. B. (1991). "Integrating speech information across talkers, gender, and sensory modality: female faces and male voices in the McGurk effect", Percept. Psychophys. Vol 50, 524536. [Crossref]
23. Hampson M., Guenther F., Cohen M., Nieto-Castanon A. (2003). Changes in the McGurk effect across phonetic contexts, Technical Report CAS/CNS 03-006, Boston University, MA, USA. Retrieved from https://pdfs.semanticscholar.org/dc98/5ea3a57d1d1175b0a4ab595e6649de409e9a.pdf.
24. Kuhl P. K., Meltzoff A. N. (1984). "The intermodal representation of speech in infants", Infant Behav. Dev. Vol 7, 361381. [Crossref]
25. Kuhl P. K., Meltzoff A. N. (1988). "Speech as an intermodal object of perception", in: Perceptual Development in Infancy: the Minnesota Symposia on Child Psychology, Vol Vol. 20, Yonas A. (Ed.), pp.  235266. Erlbaum, Hillsdale, NJ, USA.
26. Liberman A. M., Mattingly I. G. (1985). "The motor theory of speech perception revised", Cognition Vol 21, 136. [Crossref]
27. Liberman A. M., Mattingly I. G. (1989). "A specialization for speech perception", Science Vol 243, 489494. [Crossref]
28. MacDonald J. (2017). "Hearing lips and seeing voices: the origins and development of the ‘McGurk Effect’ and reflections on audio–visual speech perception over the last 40 years", Multisens. Res. DOI:10.1163/22134808-00002548.
29. MacDonald J., McGurk H. (1978). "Visual influences on speech perception processing", Percept. Psychophys. Vol 24, 253257. [Crossref]
30. Massaro D. (1987). Speech Perception by Eye and Ear: the Psychology of Lip-Reading. Erlbaum, London, UK.
31. Massaro D., Thompson L., Barron B., Laren E. (1984). "Developmental changes in visual and auditory contributions to speech perception", J. Exp. Child Psychol. Vol 41, 93113. [Crossref]
32. Massaro D. W. (1996). "Bimodal speech perception: a progress report", in: Speechreading by Humans and Machines, Stork D. G., Hennecke M. E. (Eds), pp.  79101. Springer-Verlag, Berlin, Germany. [Crossref]
33. McGurk H., Buchanan L. (1981). Bimodal speech perception: vision and hearing. Unpublished manuscript, Department of Psychology, University of Surrey, Guildford, UK.
34. McGurk H., MacDonald J. (1976). "Hearing lips and seeing voices", Nature Vol 264, 746748. [Crossref]
35. Mills A. E. (1987). "The development of phonology in the blind child", in: Hearing by Eye: the Psychology of Lip-Reading, Dodd B., Campbell R. (Eds), pp.  145161. Erlbaum, Hove, UK.
36. Mills A. E., Thiem R. (1980). "Auditory–visual fusions and illusions in speech perception", Linguist. Ber. Vol 68/80, 85108.
37. Munhall K. G., Gribble P., Sacco L., Ward M. (1996). "Temporal constraints on the McGurk effect", Percept. Psychophys. Vol 58, 351362. [Crossref]
38. Musacchia G., Sams M., Nicol T., Kraus N. (2006). "Seeing speech affects acoustic information processing in the human brainstem", Exp. Brain Res. Vol 168, 110. [Crossref]
39. Robert-Ribes J., Schwartz J.-L., Escudier P. (1995). Auditory, visual, and audiovisual vowel representations: experiments and modelling, in: Proceedings of the XIIIth International Congress of Phonetic Sciences, Vol. 3, K. Elenius and P. Branderud (Eds), pp. 114–121. ICPhS and Stockholm University, Stockholm, Sweden.
40. Robert-Ribes J., Piquemal M., Schwartz J.-L., Escudier P. (1996). "Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition", in: Speechreading by Humans and Machines, Stork D. G., Hennecke M. E. (Eds), pp.  193210. Springer-Verlag, Berlin, Germany. [Crossref]
41. Rosenblum L. D., Schmuckler M. A., Johnson J. A. (1997). "The McGurk effect in infants", Percept. Psychophys. Vol 59, 347357. [Crossref]
42. Saito D. N., Yoshimura K., Kochiyama T., Okada T., Honda M., Sadato N. (2005). "Crossmodal binding and activated attentional networks during audio–visual speech integration: a functional MRI study", Cereb. Cortex Vol 15, 17501760. [Crossref]
43. Saldaña H. M., Rosenblum L. D. (1993). "Visual influences on auditory pluck and bow judgments", Percept. Psychophys. Vol 54, 406416. [Crossref]
44. Sekiyama K. (1994). "Differences in auditory–visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility", J. Acoust. Soc. Jpn Vol 15, 143158. [Crossref]
45. Sekiyama K., Burnham D. (2008). "Impact of language on development of auditoryvisual speech perception", Dev. Sci. Vol 11, 303317. [Crossref]
46. Sekiyama K., Tohkura Y. (1991). "McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility", J. Acoust. Soc. Am. Vol 90, 17971805. [Crossref]
47. Sekiyama K., Tohkura Y. (1993). "Inter-language differences in the influence of visual cues in speech perception", J. Phon. Vol 21, 427444.
48. Sekiyama K., Braida L., Nishino K., Hayashi M., Tuyo M. (1995). The McGurk effect in Japanese and American perceivers, in: Proceedings of the XIIIth International Congress of Phonetic Sciences, Vol. 3, K. Elenius and P. Branderud (Eds), pp. 214–217. ICPhS and Stockholm University, Stockholm, Sweden.
49. Shigeno S. (2000). "Influence of vowel context on the audio–visual speech perception of voiced stop consonants", Jpn Psychol. Res. Vol 42, 155167. [Crossref]
50. Studdert-Kennedy M. (1986). "Development of the speech perceptuomotor system", in: Precursors of Early Speech, Lindblom B., Zetterstrom R. (Eds), pp.  205217. Stockton Press, New York, NY, USA. [Crossref]
51. Studdert-Kennedy M., Goodell E. W. (1992). Gestures, features and segments in early child speech, Haskins Laboratories Status Report on Speech Perception, SR-111/112, 89–102.
52. Summerfield Q. (1987). "Some preliminaries to a comprehensive account of audio–visual speech perception", in: Hearing by Eye: the Psychology of Lip-Reading, Dodd B., Campbell R. (Eds), pp.  352. Erlbaum, Hillsdale, NJ, USA.
53. Van Wassenhove V., Grant K. W., Poeppel D. (2007). "Temporal window of integration in auditory–visual speech perception", Neuropsychologia Vol 45, 598607. [Crossref]
54. Van Wassenhove V., Grant K. W., Poeppel D. (2015). "Visual speech speeds up the neural processing of auditory speech", Proc. Natl Acad. Sci. USA Vol 102, 11811186. [Crossref]
55. Werker J. F., Frost P., McGurk H. (1992). "Cross-language influences on bimodal speech perception", Can. J. Psychol. Vol 46, 551568. [Crossref]
Submit comment
Close
Comment moderation successfully completed

Sign-in

Can't access your account?
  • Key

  • Full access
  • Open Access
  • Partial/No accessInformation