of Language Instruction
David J. Francis1, Nonie Lesaux2, and Diane August3
1Texas
Institute for Measurement, Evaluation, and Statistics, University of Houston 2Graduate School of Education, Harvard University 3Center for Applied Linguistics
Texas Institute for Measurement, Evaluation, and Statistics
Language of Instruction: Overview
1. 2. 3. 4. 5. 6. Background Criteria for Study Inclusion Information on Studies Meta-analysis Methods Findings Summary and Recommendations
November 05
Background
For many years, discussion of effective instruction for Language Minority children has revolved around the question of whether and how children's first language should be used in an instructional program Language of instruction has been a pre-occupying force in the education of language minority children Surprisingly, most of the discussion has been theoretical and review oriented with limited empirical work
November 05
Background (cont)
Prior attempts to review the literature have used varying methods and selection criteria. These include 1. Baker & de Kanter (1981); Rossell & Baker (1996)a 2. Willig (1985)b; Greene (1997)b; Slavin & Cheung (2004)c; Rolstad, Mahoney, & Glass (2005)b
November 05
aVote
Counting bMeta-analytic cBest Evidence Synthesis
Background (cont)
Although prior reviews have reached different conclusions, they are not as far apart as one might imagine given the heat of the debate
1. On the one hand, some reviews have found no advantage when L1 is used in instruction 2. Other reviews, in contrast, have found advantages for the use of L1, but these have generally been in the small to moderate range (d = +.2 to +.3) 3. All reviews concur that the empirical studies in this area are lacking in various respects
November 05
Background (cont)
Problems most often cited with the empirical studies:
1. Design (inadequate control groups and length of followup; contamination; selection effects) 2. Analysis (failure to control for pre-treatment differences; failure to account for nesting) 3. Reporting (inadequate program descriptions; failure to provide information for computation of effect sizes) 4. Retrospective nature of many studies and over-reliance on data collected for other purposes
November 05
Background (cont)
At the heart of the debate lies the distinction between Bilingual and Monolingual Programs.
1. Bilingual Education (use of L1)
draws on theories of child language development emphasizes the role of language in cognition and educational attainment transfer of skills across languages hypothesizes that children learn new concepts more efficiently and with greater depth in L1, which in turn gives them a stronger foundation for future learning learning in L1 can take place as children acquire L2
November 05
Background (cont)
Several Variants of BE have been proposed and studied 1. Transitional 2. Developmental 3. Dual Language 4. Heritage Language
November 05
Background (cont)
2. Monolingual (English-Only) instruction
Theoretical underpinning is again child language development and the biological prepotency of the brain to acquire language Acquisition and mastery of language can be accelerated through immersion due to increased time on task and increased exposure Forced reliance on L2 increases its usage, i.e., creates increased practice which accelerates language learning
November 05
Background (cont)
English Only Programs
1. Students are taught in English from the beginning 2. Occasional translations or explanations in L1 may be given to support learning, but instruction is EO 3. Separate ESL classes may be included 4. Efforts may be used to scaffold instruction 5. EL learners may or may not be in classrooms with native English speakers
November 05
Searching the Literature
We systematically searched electronic databases for studies that compared some use of the native language in instruction with English-only instruction. We did not restrict the type of BE model. In addition, we attempted to obtain every study included in the reviews conducted by Willig (1985), Rossell and Baker (1996), Greene (1997), and Slavin and Cheung (2004). However, to be included in the current review, a study had to meet specific criteria.
November 05
Criteria for Study Selection
A study had to address: "What impact does the use of primary language in instruction have on the L2 literacy learning of language minority children"? 1. The subjects were language-minority students in elementary or secondary schools in Englishspeaking countries. 2. Studies of children learning a foreign language were not included. 3. Studies of instruction in heritage languages were included, if they met our other criteria (e.g., Morgan, 1971).
November 05
Criteria for Study Selection
4. Studies included at least a 6-month span between the onset of instruction and assessment of impacts 5. The study had to provide a basis for deriving expected outcomes in English literacy under both BE and EO instruction models
i.e., a suitable Control group was included and at least one measure of English literacy L1 instruction was not used in Control classrooms Case studies and descriptive studies were eliminated on this basis
November 05
Criteria for Study Selection (cont)
6. Inclusion criteria for experimental and quasi-experimental studies were the same as in the overall report Random assignment, pre-testing, or other matching criteria were used before the treatments began. Pretreatment covariates could be measures of skills related to the outcomes. No studies were excluded on the basis of level of pretreatment differences. Studies prior to 1980, tech reports, and dissertations were included to be consistent with prior reviews in this area.
November 05
Criteria for Study Selection (cont)
7. To be included in the meta-analysis, the study had to report sufficient information to compute a measure of effect size for acquisition of English literacy
At least one measure of English reading was reported Means and standard deviations were reported, or statistics were reported for which known formulae exist for converting to a measure of effect size
November 05
Criteria for Study Selection (cont)
We reviewed in narrative form some studies that did not allow the computation of effect sizes due to failed reporting, if they otherwise met criteria for inclusion We also reviewed in narrative form studies of French immersion. These were excluded in the meta-analysis because they dealt with a fundamentally different population than the EL learners of primary interest. These two sets of studies are ignored in this presentation.
November 05
Selected Studies
20 = Total Studies Reviewed (96 were identified) 16 = Studies with Language Minority Students (14 Elementary and 2 Secondary; 15 in Meta-Analysis) 5 = Studies with Language Minority Students used random assignment 26 = Total number of independent study samples in meta-analysis (Total N = 4,567; BE = 2,665; EO = 1,902) 71 = Total number of effect sizes on English literacy outcomes (Study samples by measures)
November 05
Studies with Language Minority Students
ELEMENTARY STUDENTS Lampman, 1973; Alvarez, 1975; Maldonado, 1977; Campeau et al., 1975; Maldonado, 1994; Cohen, Fathman, & Merino, 1976; Plante, 1976; Danoff, et al., 1978; Ramrez et al., 1991; de la Garza & Medina, 1985; Saldate, Mishra, & Medina, 1985; Doebler & Mardis, 19801981; Valladolid, 1991 Huzar, 1973; SECONDARY STUDENTS Kaufman, 1968
Covey, 1973;
November 05
Measures employed in Studies
California Achievement Test California Test of Basic Skills CTBS - Form S, level B CTBS - Form S, level C Durrell/Sullivan Inter-American Reading Test Iowa Test of Basic Skills Metropolitan Achievement Test Primary Acquisition of Language Test Stanford Achievement Test Stanford Diagnostic Reading Test Science Research Associates TerraNova Reading Woodcock-Johnson Wide Range Achievement Test Metropolitan Readiness Test Test of Basic Experience Unspecified
November 05
Methods
1. Once studies had been selected because they were relevant, two individuals independently reviewed them against our consistent set of standards. 2. Following these procedures, we arrived a at final set of 20 studies that diverged somewhat from those of previous reviews.
November 05
Methods (cont)
3. Included studies were coded with respect to study characteristics and treatment effects. 4. Codes and statistics for all studies included in metaanalyses were confirmed by an independent reviewer. 5. A data table was constructed in Excel and used to construct effect sizes based on Cohen's d.
November 05
Methods (cont)
X L1 - X EO ) d =( )
where )2 )2 s L1 + s EO ) = nL1 + nEO - 2
November 05
Problems in the computation of d
1. In a few instances we made assumptions to be able to estimate the effect size when information was lacking. We assumed that the pretest and posttest standard deviations were equivalent (posttest s.d. not given) We assumed that the treatment and control standard deviations were the same when only one of the two was reported. This is consistent with the assumption used when both groups were reported. In two cases we estimated the standard deviations from other studies that had used the same outcome measure at the same grades.
November 05
More Problems
None of the studies reviewed addressed the issue of non-independence of students who are nested inside instructional units. Thus, standard errors and confidence intervals around effect sizes for individual studies should be assumed to be too small. The extent of underestimation will vary across studies to an unknown degree. Consequently, we advise against interpreting the statistical significance of individual studies.
November 05
Steps to compute the average effect size
1. We treated each study sample as the unit of analysis. Thus, the 15 studies yielded 71 effect sizes across 26 samples. 2. We averaged across different reading outcomes and grades within the same study sample to derive a weighted average for that study sample. 3. We then corrected the computed d for small-sample bias by converting them to Hedges's gU 4. Each effect size was then weighted by the inverse of its variance, which varies by nL1 , nEO , and gu.
November 05
Table of Average Effect Sizes
Statistics for Each Study RCT Study Name Subgroup within Study Sample 1 Sample 1 Sample 2 Sample 1 Sample 1 Sample 1 Sample 3 Sample 2 Sample 1 Sample 1 Sample 1 Sample 2 Sample 1 Sample 1 Sample 3 Sample 2 Sample 1 Sample 1 Sample 6 Sample 1 Sample 1 Sample 8 Sample 7 Sample 3 Sample 2 Sample 5 Hedges's gU 0.0136 0.0477 0.4696 0.6583 0.7750 2.1212 -1.5981 -1.1518 -0.6052 -0.2829 -0.2621 -0.2541 -0.1863 -0.1741 0.0796 0.0947 0.1774 0.1910 0.2420 0.2541 0.3580 0.4553 0.8540 1.3929 1.8279 2.6311 Standard Error 0.2201 0.2355 0.2989 0.1555 0.4097 0.5440 0.5539 0.4591 0.1968 0.2521 0.0690 0.2389 0.2390 0.3904 0.1049 0.0954 0.1484 0.2194 0.1357 0.1441 0.1845 0.1716 0.1585 0.2628 0.2426 0.2230 Variance 0.0485 0.0555 0.0893 0.0242 0.1679 0.2959 0.3068 0.2108 0.0387 0.0636 0.0048 0.0571 0.0571 0.1524 0.0110 0.0091 0.0220 0.0482 0.0184 0.0208 0.0340 0.0294 0.0251 0.0691 0.0589 0.0497 Lower Limit -0.4178 -0.4139 -0.1161 0.3534 -0.0281 1.0550 -2.6838 -2.0516 -0.9909 -0.7770 -0.3974 -0.7224 -0.6548 -0.9392 -0.1259 -0.0923 -0.1135 -0.2391 -0.0239 -0.0283 -0.0036 0.1191 0.5434 0.8778 1.3523 2.1941 Upper Limit 0.4451 0.5092 1.0554 0.9631 1.5780 3.1874 -0.5125 -0.2519 -0.2196 0.2112 -0.1269 0.2142 0.2822 0.5911 0.2852 0.2817 0.4684 0.6211 0.5080 0.5365 0.7195 0.7916 1.1646 1.9080 2.3034 3.0681 Z-Value 0.0619 0.2025 1.5714 4.2323 1.8915 3.8992 -2.8851 -2.5087 -3.0758 -1.1223 -3.7992 -1.0634 -0.7795 -0.4459 0.7591 0.9930 1.1953 0.8703 1.7837 1.7635 1.9404 2.6540 5.3889 5.2999 7.5340 11.8001 p-Value .9506 .8396 .1161 .0000 .0586 .0001 .0039 .0121 .0021 .2617 .0001 .2876 .4357 .6557 .4478 .3207 .2320 .3841 .0745 .0778 .0523 .0080 .0000 .0000 .0000 .0000 Yes Yes Yes Yes Yes Yes Huzar, 1973 Kaufman, 1968 Kaufman, 1968 Covey, 1973 Plante, 1976 Maldonado, 1994 Cohen et al., 1976 Cohen et al., 1976 Valladolid, 1991 Saldate et al. , 1985 Danoff et al., 1978 Alvarez, 1975 Alvarez, 1975 Cohen et al., 1976 Ramrez et al., 1991 Ramrez et al., 1991 Ramrez et al., 1991 de la Garza, 1985 Campeau et al., 1975 Morgan, 1971 Maldonado, 1977 Campeau et al., 1975 Campeau et al., 1975 Campeau et al., 1975 Campeau et al.,1975 Campeau et al., 1975
Forest Plot for Individual Studies Grouped by Design Type
Group by Matching Study name Subgroup within study Outcome Time point Hedges's g and 95% CI
MD MD MD MD MD MD MD MD MD MH MH MH MH MH MHPE MHPE MS MS MS MS MSPC MSPC PC PC PCMH PCMH PCMH RS RS RS RS RS RS RS Overall
Ramirez Ramirez Campeau et al. Campeau et al. Campeau et al. Campeau et al. Campeau et al. Campeau et al. Ramirez Cohen et al. Cohen et al. Cohen et al. Danoff et al, 1977
Sample 2 Sample 3 Sample 2 Sample 3 Sample 5 Sample 6 Sample 7 Sample 8 Sample 1 Sample 1 Sample 2 Sample 3 Sample 1
Reading Total Reading Total Reading Total Reading Total Reading Total Reading Total Reading Total Reading Total Reading Total Readi...