18772787 - RESEARCH MATTERS Challenges of Value—Added...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: RESEARCH MATTERS Challenges of Value—Added Assessment Harold C. Doran and Steve Fleischman n recent years, assessment data have begun to play a pivotal role in education policy and practice. No Child Left Behind (NCLB) requires states to implement stan- dardized assessment—based systems to evaluate their schools. The NCLB approach rests on the assumption that assesstnent data can provide credible information to gauge how effectively schools and teachers are serving their students. Educators, however, recognize that because students come to school with different backgrounds, one-time assessment scores are not a fair way to compare teachers with one another when they work under vastly different circumstances. We therefore need new methods for evaluating the effectiveness of teachers and schools—methods that differ from the typical NCLB approach. The Purpose of Value-Added Assessment Value-added assessment, a statistical process for looking at test score data, is one technique that researchers have been developing to iden— tify effective and ineffective teachers and schools. In contrast to the traditional methods of measuring school effectiveness (including the adequate yearly progress system set up under NCLB), valuevadded models do not look only at current levels of student achievement. Instead, such models measure each student‘s improvement from one year to the next by l‘ollomng that student over time to obtain a gain score. The idea behind value-added modeling is to level the playing field by ttsing statistical procedures that allow direct comparisons between schools and teachers—even when those schools are working with quite different populations of students. The end result of value—added assessment is an estimate of teacher quality, referred to as a teacher (fleet in the value-added literature (Ballots, Sanders, (Sr Wright, 200%), This measure describes how well the teacher performed in improving the achievement of the students in his or her class and how this performance compares with that of other teachers, ValtteAadded models have surfaced as an important topic Value-added modeling may need to evolve into newer forms. among education policymakers, researchers, and practitioners. US. Secretary of Education Margaret Spellings has organized a federal working group to investigate how such models might be incorpo- rated into NCLB. The Government Accountability Office is investi- gating the integration of these models into state testubased accountability systems. There is also great interest in value—added assess- ment at the state level, with at least three states—Ohio, Pennsylvania, and Tennessee—using value-added assess- ment statewide. The Emerging Research Base As value—added modeling assumes a larger role in education, its research base is also flourishing. The following three topics in this field are of special interest to educators. The Complex Statistical Machinery Ever since the inception of value-added models, educators have expressed conccm that sttcli models are too statistically complex and difficult to understand (Darliugton, 1997). However. in 2004, a team of researchers at RAND brought a great deal ofclarity to the valueradded discussiOn (McCaffrey, Lockwood, Koretz, Louis, &I Hamilton, 2004i Their research documented an array of statistical approaches that can he used to analyze assessment data and discussed the benefits and limitations of each model, Some researchers have compared the results obtained from complex statistical models with those obtained from much simpler models. Tekwe and colleagues (2004) claimed that “there is little or no benefit to using the more complex model“ (p. 31), However, their study relied on a narrow data struc ture. which may have seriously limited its conclusions. Most value-added approaches remain highly technical. and there is [\bbLlLIAl'IUN‘ l'L'lk Sill‘ElH'lblUN AND CLtthlCL'LUM Dl’\'LLth.‘~1L‘NT 85 little conclusive evidence that simpler designs are just as efficient as more complex designs. Although the RAND report helped clarify the statistical methods used in value—added models, and value—added software programs are becoming more widely available (Doran & Lockwood, in press), implementing such a model remains complex. For this reason, schools and school districts that are interested in value-added modeling need to collaborate with professional organizations experienced with the challenges of this method. Test Scores and Vertical Scales In many areas of scientific research, measuring growth is straightforward. To measure changes in temperature, we need only consult a thermometer. Measuring change in student achieve- ment, however, is not as simple. For value—added modeling to work, No empirical research validates the claim that value—added models accurately identify the most effective teachers. tests must be vertically scaled (Baliou et al., 2004; Doran {81' Cohen, 2005). Essentially, vertical scaling is a statis~ tical process that connects different tests and places them on the same “ruler,” making it possible to measure growth over time. For example, one cannot measure a child’s height in inches one year and in meters the next year without adjusting the scale. To connect different tests and measure student growth. designers of value-added models commonly assume that the curriculum in higher grades is nothing more than a harder version of that in the previous grade; in other words, 8th grade math is the same as Voyage to Shelter Cove By Ralph da Costa None: This children’s book tells the story of how Serena, Herman, Pedro and Penelope find friends, solace, and sea help at Shelter Cove after the destruction of their coral reef. Voyage to Shelter Cove is the Fourth in a series of books designed to introduce school-aged children to the topic of homelessness. 32 pages | $5.00 To order, call 212-529-5252, or visit us online at www.icpny.org White Tiger Press ' New York \ Educational Publishers 86 EDUCATIONAL LEADERSHIP/NOVEMBER 2005 7th grade math, just more difficult. Therefore, one can measure a student’s increase in math knowledge by measuring his or her academic growth over time. A large body of research, however, suggests that year-to-year curricular variation is significant (Schmidt. Houang, 5r McKnight, 2005). Other researchers have demonstrated that the process used to create the vertical scales is a statistical challenge in itself and can actually introduce more error in longitudinal analyses (Doran ér Cohen, 2005‘, Michaeiides is: Haertel, 2004). These findings suggest that value- added modeling may need to evolve into newer forms. The research emerging in this area is too new, however, to allow solid conclusions. Identifying Teacher Effects Possibly the most important question about value—added assessment is whether the estimate obtained from a value-added model can actually be called a teacher effect, Can any statis tical model really sift through all the other factors that may have influenced the student's score (for example, socio— economic Status or early learning envi- ronment) and isolate the leaming that we can specifically attribute to the teacher’s methods? As it currently stands, no empirical research validates the claim that value—added models accurately identify the most effective teachers. The many anecdotal claims have not yet been verified through experimental research. Educators Take Note The research base on value-added methods is growing, and researchers are developing new approaches in an effort to intikc this technique more credible and usclul in schools. Value-added modeling is an important new area of Tt‘SCiii’L’li—fi‘tnt‘ that is playing a rapidly growing role: in shaping assessment and accountability programs. References Btilluu. D . Sanders. W.. ("\I Wright, l-‘ {20044. Controlling lair slutlcnl luck- gl‘otllid m \'fllll't'*;l(lll(‘tl .leCSSIHCnl ol tcat'licrs. journal til Ediittitiinicil and Brlicir- rural filflllifltfi. Z‘Jt ] l. 37—03 Burlington. R B. HOG?) l'lit- 'l'cnncsscc value-added assessment system. :\ dial, lciigc to familiar assessment mtthnds. In j Millmati (lid), Grading tcutlicrs. grading \1 litmls. Thousand Oaks, CA‘ Stage. DUI‘EIIL ll C... 5‘: Cohen, J (2005i The mnl’ountling cilia-t of linking bias on gains cstitiistml l'rom \‘illth‘i‘dLlClCLl models. in R Lissit: tii‘dl VLllllf-(llltlt'cl Width in (duration: 'l'litrni'r and applira- tiintx. .Vltiplc Grove. MN JAM Press Damn. H. C.‘ 5': Lockwood‘j R. tin press) Filling value-added models in R. journal nf Etlllulllimiil and Brlttiri‘nrul Sttittxti‘ts. McCallrey, D F . loclui-‘not'l. J , Kurcl; D . Louis. T. {\I Hamilton, L. ‘tZOOth Moduls l-Ul' valueiaddcrl modeling of tcziclicr ellecls juutiitil it! hlututi‘tmnl (“kl Bi‘lttti'itn‘al Strttntits. Z‘Jt l l‘ 67—101. Micl‘ttiulttlcs, M. [2&1 linerlcl, IE. H. (200+ May) Sampling it! t'niiiniuti itt‘iiis.‘ Ari itiiit'tiignizt'cl SOUHLL' 0! rum in It’s! equating t'I’vchnicnl Reportl Los Angles: Center lur the Study of Evaluation 63: National Center for Research an Evaluation, Stan- LLtrCls. and Student Testing Schmidt, W. H.. Hon-ting. R. T.. &r l\-'lcl\’niglit, C C. (200?, \r'aluc-addcd rcscarch. Right idcu lml wrung, solution7 In R Ltssn: (Ed l. Vriliir—Lttldt‘d nliltlt'lS in L‘lllitdl‘lml' Theory and L'ijlpllt'ulltlilS. Maple Grm’c. MN: jAM PrL‘SS. Tckwc. C. D , Carter. R l... Mii. (1%.. Algtnn. J . Lucas. M . Rnili.j.. at al. @0634). An empirical comparison ol statistical models for value-added assess- ment of school pct‘loi'muncc. journal (If Mutational tiiicl Bt‘ltttviurcil Sitillslltfi, 29(1 l l l—3fi. Harold C. Duran IS a Senior Research Scientist at the American institutes for Research (AtFi), Stave Fleischman, series editor of this column, is a PrinCipal Research Scientist at AIR; [email protected]' ,org. To find (jut mare. ("Ill 1-800NVORKKEY (967-5539) or VISII our website at www.workkeys.com. ZXCT l“||'|,|)l\(.. (ilHltl \'||' l'\|\|'E\'*~!|\' l‘itr mums It t|\' lIIii t'\|tt )\\| l.l'\ltl'|\"~ If! 1'5 . .11.“ ' l \iii '\|'tli\\c.i|: int» \I)\\ (II\\(.l 'Illl \‘txini Ed. D. — Educational MA. — Collaborative Leadership Bi Change (Worldwide) Educational Leadership (Culitornlu only] ' Hm l'lnp .lnLl .lp|1l_\ lx'ttdcrsllilr skills 0 ‘Illltl .1 lt'ilrl‘lil‘lj; I iimnitinily nulr VHH “litlt‘ \illl ulrtiil‘ILIL‘ ttt “lurk _ - Mi-t-i [lhl hu- “L‘L‘l‘uk'i‘ltl'w .l N'mr'ulrr ll‘fll'l‘ Ai'tt\\ltrir .ll'l\llllll’ .Il :l l'tiin' Em Ii'ur SL‘lth'NlL‘TN EILII his mtli \ntir liim lilmirlr t\p|tl\ Willi lulming All'lll i‘cwtinli ( ti 'Ilnl x tixitt' \\ Itli kirhllliguifilk'tl E.“ ull\ .IHLI .Itumiplislictl students. Ill tour I |.l\\1'(l()ll| [JIN'HH'IL‘I'IF tuur m-n' knnu lt-dgc in quH .l glitlml t1l'l\\l)l'l\ HI PHllt'NHHIHJIN .i pl’llll'\\lult;ll grtmlli pnfljnlm .intl N IIULIT f,|\‘l|[lt)l1\‘l"\ p Humility .I Hill‘ll|lll'.lll\'(‘ l'ilm'JlIUlLIl IL-mlvr “ll” {LINN In "Lilw it pin-illiu' cllJllgt‘. LlillL'i'L'tiu' I, I ’. I’scu Iim- ii ll';£LlL‘l u llU drum (“Ill Ht )1 i7} |IL HM) ll 5 l'ti\t, \\ \\ \\ .lit,'lt|iii;_t.L-tlti Await l:‘.l]i‘\ lth 51 3‘! H\l‘~l<.‘\ «\\F‘ CLRRH Lttxi Dive-Litruixt Copyright of Educational Leadership is the property of Association for Supervision & Curriculum Development. The copyright in an individual article may be maintained by the author in certain cases. Content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. ...
View Full Document

This note was uploaded on 10/07/2009 for the course EDP 351 taught by Professor East during the Spring '09 term at West Chester.

Page1 / 4

18772787 - RESEARCH MATTERS Challenges of Value—Added...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online