Measuring interlanguage pragmatic knowledge of Chinese EFL learners

Abstract

This study describes the development and validation of three test papers to test the interlanguage pragmatic knowledge of Chinese EFL learners. Three test formats - Written Discourse Completion Test (WDCT), Multiple-choice Discourse Completion Test (MDCT), and Discourse Self-assessment Test (DSAT) - and two speech acts (requests and apologies) with three sociopragmatic variables (imposition, power, and familiarity) were included in this study. The development procedure followed several steps: exemplar generation, to collect situations from the Chinese EFL learners and investigate the likelihood of each situation; metapragmatic assessment, to elicit participants' assessments of the context variables involved in each scenario; and pilot testing (both paper-and-pencil and think-aloud), to validate preliminarily the test papers. Altogether 213 subjects (both native Chinese speakers and native English speakers) participated in the development of the tests. The three test papers were administered to 200 Chinese university EFL learners who were divided into two proficiency groups according to their scores in a TOEFL test. Validation was based mainly on Messick's (1989) framework, which included a range of statistical procedures and analyses of test-taking process by means of verbal protocols. All the three methods investigated in this study were shown to have measured learners' interlanguage pragmatic knowledge of English in the Chinese EFL context, specifically in terms of the speech acts of requests and apologies. Reliability studies revealed that WDCT and DSAT were highly reliable and MDCT was also reasonably reliable. Different quantitative analyses showed that the tests tapped the intended construct and the test methods measured a similar construct. Analyses of verbal reports yielded results which supported the quantitative analyses and revealed that the construct-relevant knowledge was involved in the test takers' cognitive activities. Participants of higher level of English proficiency seemed not to have correspondingly higher interlanguage pragmatic ability. Cognitive processes involved and strategies the participants adopted in answering questions with different test methods appeared to differ. Each test method had its own advantages and disadvantages, and the practical application of the test methods depended on the purpose of the test. Metapragamtic assessment was found to be very useful in determining the sociopragmatic variables such as power, status, and familiarity between speaker and hearer.