Kanchana Prapphal Chulalongkorn University Language Institute

Abstract

This study investigates the role of the X-tests in these aspects: 1) Can they be used as an instrument to observe the ability to transfer language competence from General English to Academic English?; 2) If so, which type of grammatical competence can they measure better, syntactic competence or lexical competence; and 3) Is there any difference in the retention of grammatical competence measured by the General English X-Test and the Academic English X-Test?  Forty-eight third year Thai students enrolled in the Faculty of Science, Chulalongkorn University in 1994 participated in this study.  It was found that the X-tests were reliable and had concurrent validity with the English for Academic Achievement Tests.  The X-tests could be used as a means to examine the ability to transfer language competence from General English to Academic English.  They could assess lexical competence better than syntactic competence.  In terms of retention, the X-tests could be employed to measure the  subjects' ability to retain grammatical competence both in General English and Academic English. Introduction       The X-Test has recently been introduced in language testing research as a means to measure language proficiency.  The X-Test is a test which has been developed from the Cloze Test and the C-Test.  While the Cloze Test deletes every nth word, the C-Test deletes the second half of every second word.  Conversely, the X-Test deletes the first rather than the second half of every second word.       A few researchers have examined the X-Test in various settings.  For example, Sigott and Koberl (1993) compared the X-Test and the C-Test in assessing language proficiency of German learners.  They concluded that the X-Test was more appropriate than the C-Test when used with advanced learners because the former was more powerful in providing sufficient discrimination among subjects or tests.  Boonsothorn (1990) reported that the X-Test and the C-Test were highly reliable and valid for both L1 and L2 samples although the two tests had different factor structures.  Prapphal (1994) examined the X-Test and the C-Test with first-year science-oriented Thai university students and found that the tests were reliable and had concurrent validity with the proficiency test, the achievement test and GPA.  However, the X-Test appeared to be more closely related to the cognitive and academic skills than the C-Test.         The relationship between the cognitive and academic skills may be explained in terms of "Academic English".  Although the issue of the interrelationship between "General English" and "Academic English" is not conclusive, some studies provide explanations about the interrelationship.  Prapphal (1990) investigated three research studies in language testing dealing with "General English" and "English for Academic Purposes" conducted at Thai universities.  She pointed out that the general English tests correlated more highly with the General English Course than with the English for Academic Purposes Course in the first study.  In the second study there were direct and indirect relationships between subskills of General English and English for Academic Purposes tests.  As regards the third study, there appeared to be underlying relationships between General English and English for Academic Purposes tests.  She concluded that there may be a transfer of some language subskills across content.  Thai students seemed to transfer vocabulary, structure, and reading skills in General English to those in English for Academic Purposes.  However, the students appeared to lack the ability to transfer writing skills from General English to English for Academic Purposes.        Another aspect of the relationship between General English and Academic English can be viewed from the theory of schema.  Carrell (1984) called the background knowledge of the content area of a text "a content schema" and Hudson (1991) further proposed "technical content schema" to refer to the content from English for Specific Purposes (ESP).  Thus, different schemata might operate in different types of texts.        Although the X-Test has been studied in language testing, there is no study which examines its role as a means to measure the ability of language learners to transfer grammatical competence across content, especially Thai language learners.  Specificly, this study aims to investigate the following aspects of the X-Test:       1. Can the X-Test be used as a means to investigate the ability to transfer language competence from General English to Academic English?       2. If the X-Test can measure transferable ability, which grammatical competence can it measure better, syntactic competence or lexical competence?       3.  Is there any difference in the retention of grammatical competence as measured by the General X-Test and the Academic X-Test? Method  Subjects       Forty-eight third year Thai students enrolled in the Faculty of Science, Chulalongkorn University in 1994 participated in this study.  They took the English for Academic Purposes (EAP) Course offered by the Chulalongkorn University Language Institute.  Their major areas of studies varied.  Some studied Chemical Engineering and others studied Botany, Photography and Computer Science.  Their English proficiency also differed.  The average English grade was 2.30 while the average GPA was 2.68.  There were 19 male students and 29 female students. Instruments         Two X-tests were employed in this study.  One was adapted from the Bangkok Post, March 6, 1994 and the other was taken from New Scientist, March 6, 1993.  The content of the first X-test was considered "General English" in that it was about a bus crash (See Appendix A).  The content of the second X-test was more related to "Academic English".  It was about the ozone holes in the Arctic (See Appendix B).  Although the two tests differed in content, the number of blanks deleted was equal.  There were sixty-one deletions in each test.  The number of deleted function words and content words was quite similar.  There were eighteen function words in the General English X-Test and twenty-two in the Academic English X-Test.  As regards content words, there were forty-three in the former and thirty-nine in the latter.  The reliability coefficient of the General English X-Test, using the Cronbach alpha, was .820 and that of the Academic English X-Test was .846.  Test Administration and Scoring           The tests were administered on June 6, 1994 which was the first day of instruction, and they were given to the subjects again at the end of the semester on August 31, 1994.  The ordering effect of giving the tests was considered in the study.  The subjects who got the General English X-Test first in the pre-test got the Academic English X-Test first in the post-test and vice versa.  Each administration lasted one hour.  Scoring was carried out following the "exact word" method.  No credit was given if there was misspelling.  Each blank counted one point. Data Analysis           To answer the first research question, "Can the X-Test be used as a means to examine the ability to transfer language competence from General English to Academic English?,”  Pearson correlation coefficients were calculated using SPSSPC.  If the subjects were able to transfer knowledge of General English to Academic English, the correlation between the General English X-Test and the Academic English X-Test should be significantly and highly correlated.  Table 1 presents the means, percentage, and standard deviations of the General English X-Test, the Academic  English X-Test, the Academic English Mid-Term Achievement Test, and the Academic English Final Achievement Test.    Table 1      Means, Percentage, and Standard Deviations of the   General English X-Test (GEX), the Academic English X-Test    (AEX), the Academic English Mid-Term Achievement Test    (AEMA), and the Academic English Final Achievement Test (AEFA)   Variables  X  % SD 1 GEX (pre-test) 32.292 52.938 6.773 2. GEX (post-test)  37.833 62.021 6.336 3. AEX (pre-test) 29.396 48.190 7.615 4.  AEX (post-test) 36.813 60.349 5.603 5. AEMA 44.896  74.827 8.959 6. AEFA 26.313 52.626 8.450 N = 48   The subjects did the post-tests, both in the General English X-Test and the Academic English X-Test, better than the pre-tests.  The means of the pre-tests of the GEX and the AEX were 32.292 and 29.396 respectively.  Those of the post-tests were 37.833 and 36.813.  They did the GEX better than the AEX.  This may be due to the difficulty of the text or the subjects may have been more familiar  with the GEX dealing with a road accident than with the AEX which deals with ozone holes.  Since they had not studied the English for Academic Purposes (EAP) Course, they might have lacked "technical content schema" so they did not perform well in the pre-test.  However, after taking the EAP course, the subjects did better.  Table 2 gives the correlation coefficients of the variables.  Table 2      Correlation Coefficients of the GEX, AEX, AEMA, and AEFA    Variables GEX(pre) GEX(post) AEX(pre) AEX(post) AEMA AEFA 1 GEX (pre-test) 1.000 .765**  .674** .601** .561** .680** 2. GEX (post-test)  1.000 .530** .626** .634**  .514** 3. AEX re-test) 1.000 .476** .385* .487** 4.  AEX (post-test) 1.000 .433* .481** 5. AEMA 1.000 .706** 6. AEFA 1.000 N = 48  **p <.01 *p<  .05          The variables significantly correlated.  The highest correlation was between the GEX (pre-test) and the GEX (post-test).  The correlation coefficient was .765.  The correlation coefficient between the GEX (pre-test) and the AEX (pre-test) was .674 and that between the GEX (post-test) and the AEX (post-test) was .626.  This suggests that the subjects could transfer language competence from General English to Academic English to a certain extent although the ability to transfer is not as high as that in General English.  Thus, the X-tests might be used to observe the ability to transfer language competence across content.  However, the tests seem to measure the ability to transfer language competence in General English better than in Academic English.  This may be due to the subjects' training experience.  They had studied General English for at least eight years while they had studied Academic English for only one semester.       To answer the second research question, "Which grammatical competence can the X-tests measure better?,” a subsequent analysis was carried out.  According to Bachman (1990), grammatical competence is the knowledge of structures and lexis.  In this study, knowledge of structures was measured by the subjects' ability to fill in the blanks which were function words (prepositions, articles, conjunctions, and pronouns) on the two X-tests.  Likewise, the knowledge of lexis was assessed by the subjects' ability to complete the blanks which were content words (nouns, verbs, adjectives and adverbs) of the X-tests.  Pearson correlation was again computed to find the relationship between the syntactic competence and lexical competence of the subjects represented by the General English X-Test and Academic English X-Test.  Table 3 illustrates the results.  Table 3    Correlation Coefficients of the Studied Variables   Variables 1 2 3 4  5 6 7 8 1 SCPRGX 1.000 .400*  .587** .422* .537** .427* .279 .361* 2. LCPRGX  1.000 .450** .751** .379* .602** .400* .556** 3. SCPOGX 1.000 .582** .217 .247 .520** .506**  4.  LCPOGX 1.000 .322 .607** .317 .588**  5. SCPRAX 1.000 .493** .478** .287  6. LCPRAX 1.000 .263 .402* 7. SCPOAX 1.000 .466** 8. LCPOAX 1.000 N = 48  **p  <.01 **p<.001   1. SCPRGX = syntactic competence of the Pre-General X-Test  2. LCPRGX = lexical competence of the Pre-General X-Test  3. SCPOGX = syntactic competence of the Post-General X-Test  4. LCPOGX = lexical competence of the Post-General X-Test  5. SCPRAX = syntactic competence of the Pre-Academic X-Test  6. LCPRAX = lexical competence of the Pre-Academic X-Test  7. SCPOAX = syntactic competence of the Post-Academic X-Test  8. LCPOAX = lexical competence of the Post-Academic X-Test        As regards syntactic competence, the Pre-General X-Test correlated significantly with the Post-General X-Test and the Pre-Academic X-Test.  The correlation coefficients were .587 and .537 respectively.  However, the relationship between the Pre-General X-Test and the Post-Academic X-Test was not significant.  The correlation coefficient was .279.  This suggests that the ability to transfer syntactic competence across content might not be stable.  After being trained in the EAP course, some subjects might have developed their syntactic knowledge in Academic English.  However, the ability to transfer syntactic knowledge within the same content appears to be more stable.         Concerning lexical competence, the pattern of the relationship seems to be more consistent.  All correlation coefficients were significant.  The strongest relationship was between the Pre-General X-Test and the Post-General X-Test (r = .751).  The relationship between the Pre-Academic X-Test and the Post-Academic X-Test was not as strong as that in the General English tests.  The correlation coefficient was .402.  Again, this might be due to the effect of training.  Some subjects might have been more familiar with the academic content and had "technical content schema" so their performance in the Post-Academic X-Test was different from that in the Pre-Academic X-Test.  In conclusion, in terms of grammatical competence, the X-Tests appear to measure lexical competence better than syntactic competence, and they seem to measure lexical competence and syntactic competence within the same content better than across content.  Experimental research studies can give more revealing information about such relationships.         To examine the subjects' ability to retain grammatical competence, i.e. syntactic competence and lexical competence, when measured by the General English X-Test and the Academic English X-Test, t-tests (correlated samples) were calculated.  Table 4 shows the means, percentage, standard deviations, t-values and gained percentage of the variables.  Table 4  Means, Percentage, Standard Deviations,  T-Values and Gained Percentage of the SCPRGX, SCPOGX,  LCPRGX, LCPOGX, SCPRAX, SCPOAX, LCPRAX and LCPOAX   Variables N X % SD t gained % 1 a. SCPRGX 18 13.083  72.683 2.789 b. SCPOGX  18 14.583 81.017 1.944 -4.560*** 8.334  2. a. LCPRGX 43 19.188 44.623 5.152   b. LCPOGX 43 23.250 54.070 5.004  -7.850*** 9.447 3. a. SCPRAX 22 13.521 61.459 3.352 b. SCPOAX 22 16.229 73.763 2.434 -6.130*** 12.304 4. a. LCPRAX 39 15.896 40.759 5.348 b. LCPOAX 39 20.604 52.831 4.025 -6.220*** 12.072  N = 48  ***p < .001         The subjects improved significantly in both syntactic competence and lexical competence of General English and Academic English.  With regard to syntactic competence, the subjects improved more in Academic English than in General English.  The t-value for Academic English was -6.130 and that of General English was -4.560 and the gained percentage was 12.304% and 8.334%, respectively.  Similarly, they did better on Academic English than on General English regarding lexical competence.  The t-values were -6.220 and -7.850 and the gained percentage was 12.072% and 9.447%, respectively.  The EAP course might have helped to familiarize the subjects with "technical content schema" so they did better on both syntactic competence and lexical competence.  It is also interesting to observe that although the subjects did not study General English, they could retain grammatical competence.  However, the effect of their familiarity with the tests cannot be ruled out.  Conclusions           The X-tests as examined in this study can be used as a measure to investigate the ability to transfer language competence from General English to Academic English.  The General English X-Test and the Academic English X-Test were found to be reliable and to have concurrent validity with the Academic Achievement Tests.  As regards grammatical competence, the X-Tests appear to measure lexical competence better than syntactic competence.  The subjects seemed to transfer both lexical competence and syntactic competence within the same content better than across content.         In terms of the retention of grammatical competence, the X-Tests can be employed to assess the subjects' ability to retain both syntactic knowledge and lexical knowledge in General English and Academic English.  The training provided in an EAP course might enhance  the subjects' "technical content schema,” leading them to improve their syntactic competence as well as lexical competence when measured by the Academic English X-Test.  More research studies should be carried out to investigate the role of the X-tests in measuring the ability to transfer grammatical competence across modalities and disciplines using subjects from different backgrounds. References   Bachman, L.F. (1990).  Fundamental Considerations in Language Testing.  Oxford: Oxford University Press. Boonsathorn, S. (1990).  "C-Test/MC-Test as a Tool for Measuring English Proficiency."  Paper presented at SEAMEO Regional Language Centre on Language Testing and Language Programme Evaluation.  April 9-12, 1990 in Singapore.  Carrell, P.L. (1984). "Evidence of a Formal Schema in Second Language Comprehension.”  Language Learning.  Vol. 34, no. 2, pp. 87-112.  Hudson, T. (1991).  "A Content Comprehension Approach to Reading English for Science and Technology.”  TESOL Quarterly.  Vol. 25, no. 1, pp. 77-104.  Prapphal, K. (1990). "The Relevance of Language Testing Research in the Planning of Language Programmes."  ERIC ED 320 451.  Prapphal, K. (1994). "A Study of the C-Test and the X-Test Performed by First-Year Science-Oriented University Students.”  PASAA.  Vol. 24, December, pp. 16-23.  Sigott, G. and Koberl, J. (1993).  "Validating the X-Test.”  In Language Testing Update.  Issue 14-Autumn 1993, pp. 53-58.   Appendix A

Name: _______________________________ Registration No.:_____________ Faculty: ______________________________ Major: _____________________  GPA: ________________________________ Grade: FE I: _________________________  FE II: ______________________  _____________________________________________________________________ Read the following news report and fill in each blank with a letter.  Peru bus crash kills 44, injures 33       About 44 Easter holiday travelers were killed and 33 injured when a bus careened off a road in Peru's central Andes and plunged 400 metres down an abyss, police said on Friday.       The _ _ _ _dent occurred _ _ _ _sday night _ _ _ve Izcuchaca, a _ _wn 85 km _ _ _th of _ _is provincial _ _ _ _tal and _ _ _ut 230 km _ _st of Lima. _ _ _cue teams _ _ _ked on _ _ _day to _ _ _ _ver the _ _ _ies of _ _e victims, _ _ _ _ _ding adults _ _d children. _ _ _ _ _sses said _ _e bus _ _nt off _ _e edge _f the _ _ad after _ _ _ _ing a _ _ _ _ler truck, _ _ _ _led down _ _e side _f the _ _ _ _tain and _ _me to a  _ _lt a _ _w metres _ _om the Mantaro River. _ _ _ice did _ _t give a  _ _ _se for _ _e crash, _ _t wit- nesses _ _ _med it _n the _ _ _ _ _ _ _rating road. _ _st of _ _e passengers _ _re on _ _ _ir way _o spend _ _e long _ _ _ _day weekend _ _th family _ _ _ _ers in _ _ _er towns. _ _ _ _ _ _ _ication of _ _l victims _ _s difficult _ _ _ _use many _ _re picked _p along _ _e route _ _d not _n the  _ _ _ _ _nger list. The injured were transported by railway to a hospital in Huancayo, police said. The Bangkok Post Appendix B   Name: __________________________ Registration  No.___________________ Faculty: ________________________ Major:  __________________________ Grade: FE I: _____________________ FE II: ___________________________  ___________________________________________________________________  Read the following text and fill in each blank with a letter. Perforations appear in Arctic ozone "Mini-holes" have opened in the ozone layer over northern Europe and Canada in the past few weeks.  But, _ _ _ _ _ding to _ _e British _ _ _ _ _ctic Survey's _ _ _ne secretariat, _ _ey are "_f anything, _ _re transitory _ _d less _ _ep than _ _st year". _n recent _ _ _rs the _ _ _ne layer _ _s thinned _ _er both _ _e Arctic _ _d Antarctic _ _ch spring.  _ _t the _ _ _tic has _ _t to _ _ _fer the _ _ep, widespread _ _d sustained _ _ _ne loss _ _en each _ _ar over _ _ _ _ _ctica.  Press _ _ _ _rts last _ _ _ _end revealed _ _at, locally _ _d for _ _ _rt periods, _ _ _ne levels _n the _ _ _er atmosphere _ _ll during _ _e past _ _ _th to _ _ _ _een 10 _ _d 40 _ _ _ _ent below _ _ _mal as _ _r south _s Shetland. _ _ _ch is _ _e most _ _ _ _ _rous month _ _r the _ _ _tic ozone _ _ _er, as _ _e solar _ _ _ _ _tion that _ _ _ves ozone- _ _ _ _ _oying chemical _ _ _ _ _ions increases _ _ _er the formation of large areas of high pressure in the lower atmosphere  both lowers ozone levels, by squeezing the ozone layer above, and may provide the very cold conditions in which ozone destruction  is greatest.   New Scientist

[ INDEX ] [ TOP OF PAGE ]