Projects LEP Report 4
Relationships Between a Statewide Language Prociency Test and Academic Achievement Assessments
NATIONAL CENTER ON E D U C AT I O N A L OUTCOMES In collaboration with:
Council of Chief State School Ofcers (CCSSO) National Association of State Directors of Special Education (NASDSE)
LEP Projects Report 4
Relationships Between a Statewide Language Prociency Test and Academic Achievement Assessments
Kentaro Kato Debra Albus Kristin Liu Kamil Guven Martha Thurlow
August 2004
All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as: Kato, K., Albus, D., Liu, K., Guven, K., & Thurlow, M., (2004). Relationships between a statewide language prociency test and academic achievement assessments (LEP Projects Report 4). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
NATIONAL CENTER ON E D U C AT I O N A L OUTCOMES The project Connecting English Language Prociency, Statewide Assessments, and Classroom Performance is a subcontract (#41159) with the Minnesota Department of Education supported by a grant (#T 292B010002) from the Ofce of English Language Acquisition. Opinions expressed herein do not necessarily reect those of the Minnesota Department of Education or the U.S. Department of Education or Ofces within it.
NCEO Core Staff Deb A. Albus Ann T. Clapper Christopher J. Johnstone Jane L. Krentz Sheryl Lazarus Kristi K. Liu Jane E. Minnema Ross E. Moen Michael L. Moore Rachel F. Quenemoen Dorene L. Scott Sandra J. Thompson Martha L. Thurlow, Director
National Center on Educational Outcomes University of Minnesota 350 Elliott Hall 75 East River Road Minneapolis, MN 55455 Phone [phone] Fax [phone] http://nceo.info
The University of Minnesota is committed to the policy that all persons shall have equal access to its programs, facilities, and employment without regard to race, color, creed, religion, national origin, sex, age, marital status, disability, public assistance status, veteran status, or sexual orientation. This document is available in alternative formats upon request.
Executive Summary
Minnesota is one of many states that began development of an English prociency test before federal requirements were in place to do so. It had decided to put into place a test that would provide the state with a better and more uniform gauge of how its population of English language learners (ELLs) was doing in their acquisition of academic English language skills. Minnesota chose to adapt its test, the Test of Emerging Academic English (TEAE), from the Illinois Measure of Academic Growth in English (IMAGE). The TEAE is designed to gauge the growth of emerging academic English language skills across all grades, including three forms spanning grades 3-4, 5-6, and 7-8. The 7-8 form is also designed for use with students above grades 7-8. This report focused on state ELL performance on the TEAE, in comparison to ELL and uent English student performance on Minnesotas Comprehensive Assessment (MCA) in reading in 3rd and 5th grade, and Minnesotas Basic Skills Test (BST) in reading in 8th grade. The TEAE is designed to measure the basic English prociency required for pursuing higher-level academic achievement, while the MCA is designed to measure academic achievement toward the state standards. The Basic Skills Test in reading measures the basic skills needed to be able to graduate. Across these comparisons, our guiding research questions were to nd out what levels of the TEAE best predicts success on the MCA and BST, and whether the state decision to count as procient those ELLs who achieve at level 4 on the TEAE has a sound base of support from an assessment perspective. Study 1 addresses the questions related to the TEAE and the MCAs. Study 2 addresses the same questions for the TEAE and the BST. Key Findings: Study 1: TEAE and the MCA ELLs in TEAE level 4 are likely to do as well as native English speakers on the MCA, recognizing that there is a range of performance among native speakers. Although the specic predictive relationship (i.e., what TEAE score corresponds to what MCA score) can differ, the positive relationship between students performance on the two tests is stable across years and grades. For students with TEAE scores below about 110, there is less ability to predict MCA scores. Most students in TEAE level 3 fall into MCA levels 2A, 2B, or 3 and therefore although it is likely that many within this group score as procient (i.e., 2B or 3) others may not (2A).
Study 2: TEAE and the BST TEAE scale scores had moderate predictive power for BST performance. However, the predictability is not as good as for the MCA. To predict that a student would be likely to pass the BST, he or she must score at least 260 (i.e., achieve level 3) on the TEAE.
In conclusion, there might be stronger relationships between the MCA and 3rd and 5th grade reading skills on the TEAE because the academic language skills measured on the TEAE t those elementary grades better. Other factors besides potential discrepancies between secondary grade level skills and basic academic language skills may also account for differences in performance between the tests. These include differences in a learners age upon entering Minnesota schools, differences based on student familiarity or lack of familiarity with topical content and vocabulary for individual passages encountered on the tests, and teachers own anecdotal evidence which suggests that some students who take the TEAE do not take the test seriously. Any combination of these and other individual student factors could contribute to the TEAE not predicting success on the BST as well as on the MCA.
Overview
Minnesota is one of many states that began development of an English prociency test before federal requirements were in place to do so. It had decided to put into place a test that would provide the state with a better and more uniform gauge of how its population of English language learners (ELLs) was doing in their acquisition of academic English language skills. Minnesota chose to adapt its test, the Test of Emerging Academic English (TEAE), from the Illinois Measure of Academic Growth in English (IMAGE). The TEAE, begun before Title III legislation required an annual growth measure for English prociency under the No Child Left Behind Act of 2001, is now used to serve accountability purposes at federal and state levels, and is the ofcial measure to provide on-going identication of English language learners in Minnesota for the purpose of state funding. This said, a students procient scores on the TEAE reading and writing tests do not prohibit him or her from receiving on-going ESL/bilingual support as deemed feasible by local districts. The TEAE is designed to gauge the growth of emerging academic English language skills across all grades, including three forms spanning grades 3-4, 5-6, and 7-8. The 7-8 form is also designed for use with students above grades 7-8. Gauging growth in academic English, and even dening it, is a challenge for language acquisition specialists and assessment specialists alike. The different viewpoints on what constitutes academic English (Bailey, Butler, LaFramenta, & Ong, 2004; Chamot & OMalley, 1994; Cummins, 1979; Scarcella, 2003; Solomon & Rhodes, 1995; Stevens, Butler, & Castellon-Wellington, 2000), makes the design, implementation, and interpretation of such a prociency test complex at best, especially when translating back the results into what academic language skills a student truly needs for success across content classrooms such as reading and mathematics. This report focuses on state ELL performance on the TEAE, in comparison to ELL and uent English student performance on Minnesotas Comprehensive Assessment (MCA) in reading in 3rd and 5th grade, and Minnesotas Basic Skills Test in reading in 8th grade (BST). The TEAE is designed to measure the basic English prociency required for pursuing higher-level academic achievement, while the MCA is designed to measure academic achievement toward the state standards. The Basic Skills Test in reading measures the basic reading skills needed to be able to graduate. Across these comparisons, our guiding research questions are to nd out what levels of the TEAE best predicts success on the MCA and BST, and whether the state decision to count as procient those ELLs who achieve at level 4 on the TEAE has a sound base of support from an assessment perspective. Study 1 addresses the questions related to the TEAE and the MCAs, Study 2 addresses the same questions for the TEAE and the BST.
NCEO
1
Study 1: TEAE and MCA
Method In Study 1, we use the Minnesota state test data of third and fth graders in school year (SY) 2001-02 and 2002-03. Although the TEAE consists of reading and writing tests, we focus only on the reading test and its relationship with the MCA reading test. Hereafter, they are simply denoted by TEAE and MCA, respectively. The MCA data include test scores of all students who participated in the state assessment. The TEAE data consist of test scores of ELLs. The TEAE data originally contained 5,161 third graders and 4,688 fth graders in SY 2001-02, and 5,123 third graders and 4,683 fth graders in SY 2002-03. The MCA data originally contained 61,922 third graders and 64,408 fth graders in SY 2001-02, and 60,018 third graders and 63,350 fth graders in SY 2002-03. The data les for the same school year were merged using the student ID as the key variable. At this step, students with invalid or no student ID number were agged so that they would not be used in the subsequent analyses. The merged data were then screened to exclude students who had any missing value on variables related to test scores (i.e., raw scores, subscale scores, and scaled scores; if any of these is missing, then other scores are not reliable even if they are recorded). Students who are recorded as not tested on MCA were also excluded. The resulting sample sizes are shown in the third column in Table 1.
Table 1. Descriptive Statistics for TEAE and MCA Data TEAE Reading Scale Score SD Min 35.26 14 39.31 5 44.05 25 39.85 9 MCA Reading Scale Score SD Min 178.22 870 163.21 390 197.35 710 179.44 540
Year 01-02 02-03 01-02 02-03
Grade 3 3 5 5
N 4361 4541 3983 4238 Mean 186.22 181.94 227.94 216.60
r Max 2050 2060 2060 2220 .72 .71 .73 .73
Max 383 408 377 425
Mean 1309.11 1348.70 1334.35 1378.74
Note. N is sample size, SD is standard deviation, and r is sample correlation between TEAE and MCA.
Next, we examined the relationship between the two tests. English prociency as measured by the TEAE is considered to be prerequisite to minimal performance on the MCA. Thus, we expect that performance on the two tests is positively related, but detailed analysis will reveal more specically the degree to which they are related. We analyzed the data in three ways based on how the results of these tests may impact practice. The rst analysis examines the relationship between the two tests at the scale score level. The scale scores of the TEAE and the MCA represent English prociency and academic achievement toward the state standards, respectively. Every year performance on both tests is converted from raw scores so that they have similar distributions across years irrespective of changes in
2
NCEO
test items. Based on our research questions, we inspected scatter plots of the MCA and TEAE, and then applied regression analysis to examine the extent to which the MCA scale score is predicted by the TEAE scale score. The second analysis focused on the relationship between the two tests by the prociency or achievement level. The MCA has ve achievement levels, I, IIa, IIb, III, and IV, based on cutoff points set on the scaled score. Students who are in level IIb or above are counted as achieved for accountability purposes in Minnesota. The TEAE has four levels to represent English language prociency based on the scale score. On both the MCA and the TEAE, each level is associated with a specic description of progress toward the state standards (MCA) or English prociency (TEAE), and thus gives a brief and clearer interpretation of a test result. Also, using such levels makes the results less sensitive to measurement errors on scale scores. Examining the relationship between the two tests by the prociency or achievement level leads to relating a specic level of English prociency to a specic achievement level. The third analysis is motivated by the regulation that ELLs who have achieved the highest prociency level (level 4 on reading and level 5 writing) on the TEAE are no longer eligible for funding for LEP programs because they are regarded as having English prociency sufcient to access the academic content in the mainstream without further language support. If results of the TEAE reect this reasoning, then the distribution of MCA scores of ELLs who are in the highest English prociency level are comparable to those of students who are not ELLs. In other words, the means of the MCA score distributions of both groups of students should be almost the same and the ranges of the distributions should substantially overlap. Accordingly, the distribution of MCA scale scores for each of the TEAE prociency levels will be compared with the distribution of native English speakers. Test scores of native English speakers were taken from the Minnesota state test data as well, and those data were screened in the same manner as for the TEAE. Results Descriptive Statistics for the Entire Sample Descriptive statistics by grade and year were shown in Table 1. Within each school year, fth graders had higher mean scores on both the TEAE and MCA as expected. Fifth graders had larger variability on the MCA than third graders in both school years. Fifth graders had larger variability than third graders also on the TEAE in 2001-02, while there is little difference in 2002-03. Correlations between the TEAE and MCA are larger than .70 for all grades and years. This indicates an overall stable, positive relationship between the TEAE and MCA. Still, it is worthy of more detailed examination.
NCEO
3
Analysis of Scale Scores Scatter plots. Scatter plots of MCA scale scores and TEAE scale scores by grade and year are shown in Figures 1 through 4. These plots consistently indicate that the majority of points are positively correlated. However, there is a group of points that do not follow that major pattern in the region where TEAE scale scores are less than a given point. For third graders in 2001-02, for example, data points with TEAE scores less than about 100 seem to have almost no correlation while the majority of data points are positively correlated. For these "irregular" points, MCA scores looked highly unpredictable based on TEAE scores. Thus, it is better to separate these points in order to investigate the relationship that applies to the majority of students in the data set. The question is, however, at what point we should separate regular and irregular cases; there is no indicator variable that separates these two types of points in the data les.
Figure 1. Scatter plot of MCA and TEAE scale scores (2001-02, Grade 3)
Grade: 3
2200
2000
1800
MCA Scale Score
1600
1400
1200
1000
800
0.00
100.00
200.00
300.00
400.00
TEAE Reading Scale Score
4
NCEO
Figure 2. Scatter plot of MCA and TEAE scale scores (2002-03, Grade 3)
Grade: 3
2100.00
1800.00
MCA Scale Score
1500.00
1200.00
900.00
600.00
300.00
0
100
200
300
400
500
TEAE Reading Scale Score
Figure 3. Scatter plot of MCA and TEAE scale scores (2001-02, Grade 5)
Grade: 5
2100
1800
MCA Scale Score
1500
1200
900
600
0.00
100.00
200.00
300.00
400.00
TEAE Reading Scale Score
NCEO
5
Figure 4. Scatter plot of MCA and TEAE scale scores (2002-03, Grade 5)
Grade: 5
2500.00
2000.00
MCA Scale Score
1500.00
1000.00
500.00
0
100
200
300
400
500
TEAE Reading Scale Score
To estimate a cut off point for the scale scores for each grade and year, the following simple linear regression model is applied to the regular group of students (i.e., students with TEAE scores greater than the cutoff point) to assess the predictability of the TEAE on the MCA: MCA = (Intercept) + b1 (TEAE) + e Although there probably are multiple ways to estimate the cutoff point, a change point analysis is used for this purpose. It searches for the best cutoff point by tting two different linear regression models for regular and irregular groups, respectively. It should be noted that the TEAE scale scores show some discreteness in the score range above 300 (i.e., there are big jumps between two adjacent possible scale scores) in the score range above 300. This is more apparent for fth graders, because more students marked scores close to the maximum possible scale score. This discreteness results from the scaling, which depends on the distribution of raw scores in each grade and year. Estimation of Cutoff Scores Estimated cutoff scores are shown in the third column in Table 2. The mean squared errors of MCA scores in the irregular group estimated by the change point analysis were 167.97 and 163.53 for grade 3 (2001-02 and 2002-03, respectively), and 132.15 and 201.24 for grade 5
6 NCEO
(2001-02 and 2002-03, respectively). These are almost the same as the unconditional standard deviations listed in Table 1 except for fth graders in 2001-02. Thus, we can conclude that MCA scores of students with TEAE scores less than the cutoff points are not well predicted by the TEAE. Although these cutoff points vary across years and grades, the unpredictability is likely to occur when the TEAE score is less than about 110.
Table 2. Estimates of cutoff scores and regression coefcients Year 01-02 02-03 01-02 02-03 Grade 3 3 5 5 Cutoff 124.87 114.39 131.93 130.71 N 4217 4417 3953 4161 Intercept (b0) 501.46 739.63 561.85 593.82 Slope (b1) 4.29 3.31 3.38 3.60 R2 .58 .54 .54 .56
Note. Intercepts and slopes are for the regular group of students with TEAE scores greater than the cutoff point. N is the number of students included in the regular group, and R2 is the squared multiple correlation.
Regression Analysis for the Regular Group In the fourth through seventh columns in Table 2 are shown the number of students in the regular group, estimated intercept, slope, and R squared for the regular group of students (i.e., students with TEAE scores greater than the cutoff point). The slopes range from 3.31 to 4.29, and the corresponding R2s range from .54 to .58. These results indicate that more than 54% of variation of the MCA scale score can be accounted for by the TEAE scale score for the regular group of students. This is a strong positive relationship. The results also indicate, however, that slopes vary to some extent across years and grades. The estimated regression lines are plotted in Figure 5. As the slope estimates indicate, the lines are almost parallel except for grade 3 in 2001-02, where the regression line is slightly steeper than the others. Also, vertical locations of the lines vary in the 200 range for the MCA score scale. The lines for grade 3 are higher than those for grade 5 in Figure 5, but more longitudinal data would be required to infer systematic effects of grade levels on regression lines. Overall, although the specic predictive relationship (i.e., what TEAE score corresponds to what MCA score) can differ, the positive relationship between the two tests is stable across years and grades. Thus, we expect that increased English prociency is associated with progress toward the state academic standards.
NCEO
7
Figure 5. Comparison of Estimated Regression Lines
2500
MCA Reading Scale Score
2000 Grade 3:2001-02 Grade 3:2002-03 Grade 5:2001-02 1000 Grade 5:2002-03
1500
500
0 0 100 200 300 400
TEAE Reading Scale Score
Relationship by Prociency or Achievement Level Grade 3 TEAE Level and MCA Level Correspondence Tables 3 and 4 show the number of third graders cross-classied by TEAE prociency levels and MCA achievement levels in 2001-02 and 2002-03. Level 1 of the TEAE includes the irregular group of students found in the analysis of scale scores. Both 2001-02 and 2002-03 results consistently indicated the following. First, students in TEAE level 1 are likely (about 80%) to be in level 1 on the MCA, and thus to be counted as "not procient" for accountability purposes. This is a clear indication that basic English prociency is a prerequisite to achieving higher-level academic reading skills. Second, students in TEAE level 4 are likely to achieve level 3 or 4 on MCA, and thus to be counted as procient for accountability purposes (the result for 2001-02 may not be reliable due to the small sample size of 24 in TEAE level 4). Thus, procient English learners can do well on the MCA. Finally, TEAE levels 2 and 3 seem to have no single corresponding level on the MCA. Most students in TEAE level 2 fall in MCA level 1, 2A, or possibly 2B, although they are unlikely to be procient (2B) on the MCA. Also, most students in TEAE level 3 fall into MCA levels 2A, 2B, or 3. They are likely to be procient on the MCA but there is still some possibility that they would not be procient. Although there is no clear one-to-one correspondence between the TEAE prociency levels and the MCA achievement levels, ELLs who are in TEAE level 3 or 4 are likely to be procient (i.e., scoring in level 2B or above) on the MCA.
8
NCEO
Table 3. Correspondence between TEAE Prociency Levels and MCA Achievement Levels (2001-02, Grade 3)
MCA Reading Achievement Level 1 2A 2B 3 4 1406 274 51 23 1 80.1 15.6 2.9 1.3 0.1 73.1 23.2 7.7 4.5 1.1 32.2 6.3 1.2 0.5 0.0 515 864 519 311 25 23.1 38.7 23.2 13.9 1.1 26.8 73.1 78.6 61.2 28.7 11.8 19.8 11.9 7.1 0.6 3 43 89 167 46 0.9 12.4 25.6 48.0 13.2 0.2 3.6 13.5 32.9 52.9 0.1 1.0 2.0 3.8 1.1 0 1 1 7 15 0.0 4.2 4.2 29.2 62.5 0.0 0.1 0.2 1.4 17.2 0.0 0.0 0.0 0.2 0.3 1924 1182 660 508 87 44.1 27.1 15.1 11.6 2.0 100.0 100.0 100.0 100.0 100.0 44.1 27.1 15.1 11.6 2.0 Total 1755 100.0 40.2 40.2 2234 100.0 51.2 51.2 348 100.0 8.0 8.0 24 100.0 0.6 0.6 4361 100.0 100.0 100.0
TEAE Reading Proficiency Level
1
2
3
4
Total
Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total%
Note. Dark gray cells indicate that the row proportion is larger than 50% (i.e., more than 50% of MCA scores were at this level or these levels when the TEAE score was the one in the row), and light gray cells indicate that the row proportion is larger than 20% (i.e., more than 20% of MCA scores were at this level or these levels when the TEAE score was the one in the row).
Grade 5 TEAE Level and MCA Level Correspondence Results are shown in Tables 5 (for the 2001-02 data) and 6 (for the 2002-03 data). Fifth graders showed results similar to those of third graders for both academic years. There is a clearer indication than for third graders that TEAE level 4 corresponds to MCA level 3. Also, TEAE level 2 corresponds to MCA levels 1 or 2A, and TEAE level 3 to MCA levels 2A, 2B, or 3. These observations are consistent in both school years. Again, we can conclude that increased English prociency of English learners is associated with higher performance on accountability measures.
NCEO
9
Table 4: Correspondence between the TEAE Prociency Levels and the MCA Achievement Levels (2002-03, Grade 3)
MCA Reading Achievement Level 2A 2B 3 4 323 58 21 4 0 79.6 14.3 5.2 1.0 0.0 21.7 4.4 2.4 0.5 0.0 7.1 1.3 0.5 0.1 0.0 1081 747 255 82 4 49.8 34.4 11.8 3.8 0.2 72.6 57.0 28.7 10.5 5.2 23.8 16.5 5.6 1.8 0.1 84 495 561 523 29 5.0 29.3 33.2 30.9 1.7 5.6 37.8 63.2 67.2 37.7 1.8 10.9 12.4 11.5 0.6 1 10 50 169 44 0.4 3.6 18.2 61.7 16.1 0.1 0.8 5.6 21.7 57.1 0.0 0.2 1.1 3.7 1.0 1489 1310 887 778 77 32.8 28.8 19.5 17.1 1.7 100.0 100.0 100.0 100.0 100.0 32.8 28.8 19.5 17.1 1.7 1 Total 406 100.0 8.9 8.9 2169 100.0 47.8 47.8 1692 100.0 37.3 37.3 274 100.0 6.0 6.0 4541 100.0 100.0 100.0
TEAE Reading Proficiency Level
1
2
3
4
Total
Count Row% Column% Total% Count Row% Column% Total% Row% Count Column% Total% Count Row% Column% Total% Count Row% Column% Total%
Note. Dark gray cells indicate that the row proportion is larger than 50% (i.e., more than 50% of MCA scores were at this level or these levels when the TEAE score was the one in the row), and light gray cells indicate that the row proportion is larger than 20% (i.e., more than 20% of MCA scores were at this level or these levels when the TEAE score was the one in the row).
Table 5. Correspondence between TEAE Prociency Levels and MCA Achievement Levels (2001-02, Grade 5)
MCA Reading Achievement Level 2A 2B 3 4 362 24 3 2 0 92.6 6.1 0.8 0.5 0.0 24.5 2.0 0.6 0.3 0.0 9.1 0.6 0.1 0.1 0.0 1002 672 126 99 6 52.6 35.3 6.6 5.2 0.3 67.8 56.2 27.1 14.2 4.1 25.2 16.9 3.2 2.5 0.2 111 451 280 393 61 8.6 34.8 21.6 30.3 4.7 7.5 37.7 60.2 56.2 41.8 2.8 11.3 7.0 9.9 1.5 2 49 56 205 79 0.5 12.5 14.3 52.4 20.2 0.1 4.1 12.0 29.3 54.1 0.1 1.2 1.4 5.1 2.0 1477 1196 465 699 146 37.1 30.0 11.7 17.5 3.7 100.0 100.0 100.0 100.0 100.0 37.1 30.0 11.7 17.5 3.7 1 Total 391 100.0 9.8 9.8 1905 100.0 47.8 47.8 1296 100.0 32.5 32.5 391 100.0 9.8 9.8 3983 100.0 100.0 100.0
TEAE Reading Proficiency Level
1
2
3
4
Total
Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total%
Note. Dark gray cells indicate that the row proportion is larger than 50% (i.e., more than 50% of MCA scores were at this level or these levels when the TEAE score was the one in the row), and light gray cells indicate that the row proportion is larger than 20% (i.e., more than 20% of MCA scores were at this level or these levels when the TEAE score was the one in the row).
10
NCEO
Table 6. Correspondence between TEAE Prociency Levels and MCA Achievement Levels (2002-03, Grade 5)
1 TEAE Reading Proficiency Level 1 Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total% Count Row% Column% Total%
2
3
4
Total
MCA Reading Achievement Level 2A 2B 3 4 411 63 10 14 1 82.4 12.6 2.0 2.8 0.2 41.2 4.7 1.2 1.5 0.7 9.7 1.5 0.2 0.3 0.0 458 531 108 55 0 39.8 46.1 9.4 4.8 0.0 45.9 39.4 13.2 5.9 0.0 10.8 12.5 2.5 1.3 0.0 125 723 643 577 58 5.9 34.0 30.2 27.1 2.7 12.5 53.7 78.6 62.0 40.0 2.9 17.1 15.2 13.6 1.4 3 30 57 285 86 0.7 6.5 12.4 61.8 18.7 0.3 2.2 7.0 30.6 59.3 0.1 0.7 1.3 6.7 2.0 997 1347 818 931 145 23.5 31.8 19.3 22.0 3.4 100.0 100.0 100.0 100.0 100.0 23.5 31.8 19.3 22.0 3.4
Total 499 100.0 11.8 11.8 1152 100.0 27.2 27.2 2126 100.0 50.2 50.2 461 100.0 10.9 10.9 4238 100.0 100.0 100.0
Note. Dark gray cells indicate that the row proportion is larger than 50% (i.e., more than 50% of MCA scores were at this level or these levels when the TEAE score was the one in the row), and light gray cells indicate that the row proportion is larger than 20% (i.e., more than 20% of MCA scores were at this level or these levels when the TEAE score was the one in the row).
NCEO
11
Comparability of MCA Scores Grade 3 TEAE Level by MCA Scale Scores Mean MCA scale scores by TEAE prociency level were compared with the mean MCA scale score of native English speakers, and similar comparisons were made for dispersion of test scores (see Table 7). Also, boxplots were drawn (see Figures 6 and 7). In these, the box represents the middle 50% of the data, the top line represents the 75th percentile and the bottom line represents the 25th percentile. A line segment in the box indicates the median. The length of whiskers outside the box is usually taken 1.5 times as large as the interquartile range, which is the height of the box. All values outside the range of the whiskers are marked as outliers and represented as dots in the plot. As in the comparison by the prociency or achievement levels, the irregular group of students was included in the data. In the subsequent tables and gures, the group of native English speakers is designated as "No TEAE."
Table 7. Mean MCA Scale Score by TEAE Prociency Level (Grade 3)
TEAE Level 1 TEAE Level 2 TEAE Level 3 TEAE Level 4 No TEAE
Mean 1176.83 1374.23 1544.89 1711.67 1500.56
2001-02 SD 134.23 129.56 132.37 152.53 201.61
N 1803 2246 352 24 54263
Mean 1169.46 1275.52 1448.44 1575.96 1531.90
2002-03 SD 140.41 129.68 103.66 106.47 180.25
N 429 2174 1694 275 53556
Note. The group of native English speakers is designated as No TEAE.
In the 2001-02 school year, ELLs in TEAE levels 3 and 4 had higher mean scores than native English speakers. The result for TEAE level 4, however, is not reliable due to the small sample size; the mean and standard deviation for that group are both too high. Dispersion of scores is almost the same for all TEAE prociency levels except for TEAE level 4, and they are much smaller than the dispersion for No TEAE. This is a natural result because TEAE levels are correlated to the MCA scale scores. Figure 6 shows that the ranges indicated by whiskers (i.e., the lines extending from the box) for TEAE levels 2, 3, and 4 are completely within the whisker range of No TEAE (and the interquartile ranges of these levels indicated by boxes also substantially overlap that of No TEAE). Yet, the location of the distribution of TEAE level 2 is substantially lower compared with No TEAE. These results indicate that ELLs in TEAE levels 3 or 4 can perform as well on the MCA as native English speakers. In SY 2002-03, the pattern of score distributions is somewhat different from that of SY 2001-02. The mean score in 2002-03 is lower than in 2001-02 at each TEAE prociency level, whereas
12 NCEO
the mean score of No TEAE in 2002-03 is higher than in 2001-02 (see Table 7). Also, the score dispersion tends to be smaller as the TEAE level goes up, unlike in 2001-02. TEAE level 4 has a higher mean score than No TEAE as well as in 2001-02, but TEAE level 3 does not. Figure 7 shows that the score distributions of TEAE levels 3 and 4 are completely within the range of No TEAE, but the distribution of TEAE level 3 is located relatively low to that of No TEAE with little overlap of the interquartile range. Thus, the 2002-03 data indicate that while ELLs in TEAE level 4 can perform as well on the MCA as native English speakers, this may not be the case for those in TEAE level 3.
Figure 6. Boxplots of Mean MCA Scale Scores by TEAE Prociency Level (2001-02, Grade 3)
131,497 111,874
2000
110,974 123,202
67,121 113,123 67,177 114,024 129,094 129,541 70,950 112,723 71,420 67,946 120,420
121,733 67,110 103,761 93,216 121,253 124,573
114,006 110,971
MCA Scale Score
1500
1000
82,580 67,436
82,371 117,700 67,150
68,165 67,216 67,111 67,187
131,736 131,527 131,745 131,458 129,835 85,085
500
131,459
TEAE Level 1
TEAE Level 2
TEAE Level 3
TEAE Level 4
No TEAE
TEAE Proficiency Level / No TEAE
NCEO
13
Figure 7. Boxplots of Mean MCA Scale Scores by TEAE Prociency Level (2002-03, Grade 3)
2100.00
7,386
38,369 34,105
60,111
1800.00
6,851 55,265 12,097 42,480
48,275 50,503
MCA Scale Score
1500.00
1,648 15,865
1200.00
42,028 42,306 13,631
900.00
1,551
412 60,002 59,946 59,928 59,482 51,976 59,419 49,896 523 56,263 30,369 30,381 36,827 24,677
600.00
20,677
12,560
300.00
TEAE Level 1
TEAE Level 2
TEAE Level 3
TEAE Level 4
No TEAE
TEAE Proficiency Level / No TEAE
Grade 5 TEAE Level by MCA Scale Scores A summary of the MCA scale scores by TEAE prociency level is shown in Table 8, and boxplots are shown in Figures 8 and 9. Fifth graders in both 2001-02 and 2002-03 school years consistently show a distributional pattern similar to third graders in 2001-02. In each school year, the distribution of MCA scale scores of TEAE level 4 has almost the same mean as the No TEAE group, and the range of the distribution is completely within that of the No TEAE group. The range of TEAE level 3 is also within that of No TEAE, but its mean is substantially lower than that of No TEAE in both school years. Thus, for fth graders, students in TEAE level 4 are comparable to native English speakers.
Table 8. Mean MCA Scale Score by TEAE Prociency Level (Grade 5)
TEAE Level 1 TEAE Level 2 TEAE Level 3 TEAE Level 4 No TEAE
2001-02 Mean SD 1092.51 170.12 1265.09 139.54 1445.88 144.26 1575.86 150.46 1567.84 211.23
N 438 1934 1315 399 57147
Mean 1145.44 1284.62 1440.35 1593.39 1580.66
2002-03 SD 156.01 118.88 125.87 134.78 196.60
N 515 1157 2137 463 57104
14
NCEO
Figure 8. Boxplots of Mean MCA Scale Scores by TEAE Prociency Level (2001-02, Grade 5)
131,497 111,874
2000
110,974 123,202
67,121 113,123 67,177 114,024 129,094 129,541 70,950 112,723 71,420 67,946 120,420
121,733 67,110 103,761 93,216 121,253 124,573
114,006 110,971
MCA Scale Score
1500
1000
82,580 67,436
82,371 117,700 67,150
68,165 67,216 67,111 67,187
131,736 131,527 131,745 131,458 129,835 85,085
500
131,459
TEAE Level 1
TEAE Level 2
TEAE Level 3
TEAE Level 4
No TEAE
TEAE Proficiency Level / No TEAE
Figure 9. Boxplots of Mean MCA Scale Scores by TEAE Prociency Level (2002-03, Grade 5)
65,177
110,708 125,414
128,696
2000.00
102,796 128,058 95,857 69,935 111,041 110,865 110,140
119,234
MCA Scale Score
1500.00
126,413 65,816 117,415 88,309 98,500 110,736 110,691 93,369
500.00
1000.00
128,313 128,744 128,488 98,867 128,356 107,398 108,990 108,944 65,590 97,621 67,773 67,860 128,496
TEAE Level 1
TEAE Level 2
TEAE Level 3
TEAE Level 4
No TEAE
TEAE Proficiency Level / No TEAE
NCEO
15
Study 2: TEAE and BST
Method The data used in these analyses, like those used for the MCA analyses, are from 2001-02 and 2002-03. With state eighth grade performance, we again focus on the TEAE reading test in comparison to the BST reading test (hereafter, referred to as TEAE and BST). The TEAE data originally contained 4,019 eighth graders in SY 2001-02, and 3,865 in SY 2002-03. The BST data originally contained 61,922 eighth graders, 66,769 in SY 2001-02, and 66,975 in SY 200203. The data were screened in the same manner as for the TEAE-MCA analysis: excluding students who (a) had any missing value on variables related to test scores, (b) were recorded as not tested on the BST, and (c) had the minimum possible score on the TEAE. The resulting sample sizes are shown in Table 9.
Table 9. Descriptive Statistics for TEAE-BST Data Year 01-02 02-03 Grade 8 8 N 3315 3331 TEAE Reading Scale Score Mean SD Min 247.77 42.19 59 243.87 39.00 28 BST Reading Scale Score SD Min 42.62 434 44.04 456 r Max 750 750 .71 .66
Max 417 437
Mean 589.89 585.50
Note. N is the sample size, SD is the standard deviation, and r is the sample correlation between TEAE and BST.
The purpose of this analysis was to examine how basic English prociency measured by the TEAE relates to (a) acquisition of basic academic reading skills, and (b) the reading skills needed to pass the BST as needed for graduation. We therefore analyzed the data in two ways. The rst analysis examined how English prociency affects acquiring basic academic skills. This was done by examining the relationship of the TEAE and BST at the scale score level. We used scatterplots and regression analysis to examine the extent to which the BST scale score is predicted by the TEAE scale score. The second analysis focused on how English prociency affects passing rates. The criterion for graduation is a BST scale score greater than 600. In this analysis, the TEAE scale score is used as a predictor of the passing rate. Since the criterion variable for each student is binary (i.e., passed or failed), the logistic regression analysis is employed to predict passing rates. Passing rates are also compared across prociency levels of the TEAE. Results Descriptive Statistics for the Entire Sample Means and standard deviations of test scores are very similar in both academic years. The correlations between the TEAE and BST were .71 and .66 in SY 2001-02 and SY 2002-03,
16 NCEO
respectively. They are slightly smaller than the correlations between the MCA and TEAE. Also, the correlation in 2002-03 is smaller than that in 2001-02. Analysis of Scale Scores Scatter plots. BST scale scores were plotted against TEAE scale scores for each school year. These plots, however, showed that the BST scale score increases exponentially rather than linearly as the TEAE scale score increases. This seemed to be a result of scaling of the BST scale score. The distribution of the BST raw scores peaked close to the maximum possible score. Then, on the resulting scale, raw score points close to the maximum were stretched out, that is, intervals between these scale scores were much longer than those between scale scores from lower raw scores. In order to apply linear regression models, the BST scale score was log-transformed so that the relationship between the BST and TEAE was more linear. The resulting scatter plots for 2001-02 and 2002-03 school years are shown in Figures 10 and 11. Figures 10 and 11 show that the TEAE scale score and the log-transformed BST scale score is positively related, and the relationship is almost linear. However, we do observe a small number of data points that lie outside the central region in which most of the data points concentrate. These observations will negatively affect the predictability of the BST scale score. Unlike the MCA, these points are distributed across almost the entire range of the TEAE scale score. Also, higher TEAE or BST scores show discreteness due to the scaling, although the discreteness of the BST has been weakened by the log-transformation. With the information currently at hand, we have no basis for removing these data points. Thus, all of these data points were used for the regression analysis. Regression Analysis The following linear regression model, was applied by school years in order to assess the predictability of the BST: log(BST) = (Intercept) + b1(TEAE) + e The results are shown in Table 10. The estimated regression lines are almost the same, but R2 for 2002-03 is smaller than for 2001-02. Also, these R2s are smaller than those for the MCA. Thus, the BST scale score can be predicted by the TEAE scale score to a moderate degree, because English prociency affects acquiring basic academic skills in reading. However, the predictability is not as good as for the MCA. The relationship between the BST and TEAE seems to be stable across years, as shown in Figure 12 in which the estimated regression curves from both school years are plotted (the log-BST scale score is transformed back to the original scale score). TEAE scores that give the predicted value of the BST score of 600 (...