SuttonTrustFullReportFinal11.pdf - Evidence on the effects...

Unformatted text preview: Evidence on the effects of selective educational systems A report for the Sutton Trust Robert Coe, Karen Jones, Jeff Searle, Dimitra Kokotsaki, Azlina Mohd Kosnin, Paul Skinner CEM Centre, Durham University, UK October 2008 Executive summary The context of selection 1. There are 164 secondary maintained selective (grammar) schools in England, located in 36 Local Authorities (LAs). They arise from a complex history, and exist in the context of a wide variety of different kinds of secondary school (Chapter 1). 2. A number of arguments have been made for selection. These include the claims that it is appropriate for different types of pupil to have different kinds of education; that teaching can best be targeted at a narrow ability range; that grammar schools are meritocratic and socially redistributive by providing advantage for the bright but poor; that they are socially inclusive, as they keep the middle classes in state education; that the academic elite should be a priority for education; that grammar schools provide a beacon of excellence; that they achieve better academic results; and that selection operates elsewhere within the educational system (Section 2.1, p15). 3. Arguments against selection include the claims that selection tests are never fair or adequate; that ability is multi-dimensional and fluid; that the impact of failure on pupils not selected is unacceptable; that selection has an adverse effect on the primary school curriculum; that it is socially divisive; that selection compounds disadvantage; that it is the socially disadvantaged who should be a priority for education; that selection limits parental choice; and that selective systems produce worse academic results (Section 2.2, p20). Existing studies 4. A number of studies have previously tried to compare the performance of pupils in selective and non-selective schools. Two major contributions in the 1980s (Steedman, 1980, 1983 and Marks et al., 1983, 1985) were followed by more recent interest (Jesson, 2000, 2001; Prais, 2001, Yang and Woodhouse, 2001). The advent of national pupil-level datasets allowed Schagen and Schagen (2003, 2005) and Atkinson et al. (2004) to advance our knowledge appreciably. Recent updates using data from the 1958 birth cohort National Child Development Study (NCDS) (Sullivan and Heath, 2002; Galindo-Rueda and Vignoles, 2004; Manning and Pischke, 2004) have also contributed. Some studies of Northern Ireland, the UK as a whole, and Australia are also relevant (Chapter 3). 5. Most of these studies suffer from limitations of methodology, data or interpretation; some are quite serious. In particular, their inability to control for other differences; problems with the quality of baseline or outcome data; issues in the calculation of value-added; inappropriate choice of the unit of analysis; failure to acknowledge the heterogeneity of selective systems; focus on cohorts that were educated in the 1970s; and researchers’ apparent preconceptions all undermine the trustworthiness of their results (Chapter 4, p107). iii EFFECTS OF SELECTIVE EDUCATIONAL SYSTEMS 6. Most of the existing studies report somewhat mixed results, with no clear advantage to either selective or non-selective systems as a whole. However, the majority of studies (and all of those we judge to be methodologically strongest) report that pupils who attend grammar schools do better than equally able pupils in comprehensives. This is true both for those that used national datasets and those based on the NCDS data. Some studies identify particular subgroups as benefiting most from attending a grammar school (Section 5.4, p131). Our own analysis of national datasets 7. Just under 4% of 11-16 year-olds attend grammar schools. As well as being more able, they are also significantly less likely to be eligible for Free School Meals (FSM) than those in non-selective schools (Section 6.1.1, p137). This difference does not seem to be fully explained by their higher ability or their tendency to live in more socially advantaged areas (Section 7.2.9, p183). 8. In terms of school-level characteristics, grammar schools are very different from other schools. All have sixth forms, compared with about half of non-grammar schools. Fewer than 10% of non-grammars are single-sex schools, compared to three-quarters of grammars. Grammar schools contain higher proportions of Specialist schools and Foundation schools (Section 6.1.2, p138). 9. Pupils who attend grammar schools do not all live in the LA of the school they attend. Nationally, about 20% of grammar school pupils come from outside the LA; for some LAs, this figure is as high as 75% (Section 6.3.1, p142). Some 80 LAs have more than 1% of the pupils who live in their area attending grammar schools, compared with only 36 LAs that actually have grammar schools of their own (Section 6.2.2, p139). Across England as a whole, one third of the wards in the country (33%) house at least one pupil who attends a grammar school (Section 6.2.1, p139). The concept of a ‘selective’ LA, whose performance can be isolated, is therefore rather problematic. 10. Different qualifications taken at KS4 are not of equal difficulty, and the points awarded to ‘equivalent’ qualifications do not necessarily reflect this. In some subjects, students are systematically getting better grades than those same students do across their other subjects. Overall, more able pupils tend to take harder qualifications, and those in grammar schools even harder still. Any comparison of grades achieved should therefore take account of these differences (Section 6.4, p144). 11. We have developed a way of defining the ‘creaming’ effect of any given grammar school on each non-grammar school (Section 7.1, p153). A relatively small number of schools are substantially creamed: 161 schools (5% of non-selective schools nationally) lose more than 20% of their potential pupils to grammar schools. Three-quarters of these schools are in just four LAs. Just under one-third of the non-selective schools in the country (32%) lose between 0 and 1% of the pupils they might have had, with a further third (35%) losing between 1% and 20%. Throughout the country as a whole only about one-quarter of non-selective schools (28%) lose no pupils at all to grammar schools (p157). This far-reaching but lowlevel impact of selection is very different from the traditional picture of iv EXECUTIVE SUMMARY self-contained ‘selective’ and ‘comprehensive’ areas, with grammar and secondary modern schools on the one hand, and comprehensive schools on the other. 12. We have also developed a method for calculating ‘selectivity’: the extent to which schools discriminate academically and socially between the pupils they take and those, living in the same neighbourhoods, whom they do not (Section 7.2, p161). Not surprisingly, grammar schools are substantially more academically selective than other schools, though, surprisingly, there is actually some overlap. Grammar schools are also more socially selective than other schools, but here the overlap is much bigger; the most socially selective state schools in the country are ‘nonselective’ schools. These socially selective schools are more likely to be Voluntary Aided or CTCs, to be single sex, faith schools, larger than average and drawing from more competed wards (p178). 13. In comparing the performance of pupils in selective and non-selective schools, a number of choices have to be made. These include how the different Key Stage 4 outcomes should be treated; what kinds of factors should be taken into account in order to make comparisons fair; what kinds of statistical models should be used; and which groups should be compared. Implications of different choices are considered (Section 8.1, p187). 14. In terms of raw KS4 (GCSE) results, it is clear that pupils in grammar schools do much better. This advantage remains, although the difference is smaller, if consideration is limited to pupils who achieved level 5 or higher in each of mathematics, English and science at KS2 (Section 8.2, p195). 15. Regression and multilevel analyses were conducted on the national pupil data (section 8.3, p209). Propensity Score Matching was also applied (section 8.5, p222). Most of these analyses suggest that pupils in grammar schools do a little better than similar pupils in other schools, with the difference somewhere between zero and three-quarters of a GCSE grade per subject. In general, the more factors introduced into the model, the smaller the difference. In particular, the inclusion of school composition variables reduces the grammar school advantage considerably (p213). The choice of different statistical models also makes a difference to the outcome, as does the use of different outcome measures. On the other hand, the choice of different comparison groups does not seem to make much difference to the results. The schools that are affected by grammar schools, in terms of losing pupils to them, are performing no differently from all other schools (Section 8.3.4, 215). Although these analyses indicate that grammar school pupils appear to make greater progress from KS2 to KS4 than other pupils, we also find that these same pupils were already making more progress from KS1 to KS2 (ie in their primary school). This suggests that there may be important but unmeasured differences between grammar and non-grammar school pupils and somewhat undermines our confidence in these estimates of a ‘grammar school effect’ (section 8.4, p220). 16. Overall, therefore, we find that although many of our analyses identify a small positive advantage in GCSE achievement for pupils at grammar schools, there are good reasons to be cautious of describing this as a grammar school ‘effect’. At least a part of this difference is likely to be a v EFFECTS OF SELECTIVE EDUCATIONAL SYSTEMS result of inadequate data and bias in the evaluation designs available to us. vi Contents Part I Introduction ............................................................................................... 1 1. Introduction: The context of selection........................................................ 3 1.1. The context of selection in England .......................................................................................3 1.1.1. A brief history of English schools............................................................................. 3 1.1.2. Different schools and their admissions policies ................................................... 8 1.1.3. Grammar schools today ............................................................................................. 10 1.2. Overview of Report ................................................................................................................13 2. Arguments for and against selection .........................................................15 2.1. The case for selection ..............................................................................................................15 2.1.1. Appropria te (different) education for different types of pupil ...................... 15 2.1.2. Teach ing is better targeted at narrow ability range.......................................... 17 2.1.3. Grammar schools are meritocratic and socially redistributive by providing advantage for the bright but poor....................................................... 17 2.1.4. Socia lly inclusive, as it keeps the middle classes in sta te education ........... 17 2.1.5. Academic elite should be a priority for education ............................................. 18 2.1.6. Grammar schools are a beacon of excellence......................................................... 18 2.1.7. Better academic results ............................................................................................. 18 2.1.8. Selection operates elsewhere with in the educational system........................ 19 2.2. The case against selection ......................................................................................................20 2.2.1. Selection tests are never fa ir or adequate ............................................................. 20 2.2.2. Ability is multi-dimensional and fluid ................................................................ 21 2.2.3. Impact of fa ilure on pupils not selected................................................................. 22 2.2.4. Adverse effect on primary school curriculum....................................................... 22 2.2.5. Socia lly divisive ........................................................................................................ 23 2.2.6. Selection compounds disadvantage ........................................................................ 23 2.2.7. Socia lly disadvantaged should be a priority for education ............................ 23 2.2.8. Selection limits parenta l choice ............................................................................. 24 2.2.9. Worse academic results ............................................................................................. 24 Part II Literature Review ....................................................................................25 3. Existing Empirical Evidence on the Effects of Selective Systems............27 3.1. Major studies in the 1980s....................................................................................................27 3.1.1. Steedman (1980): Progress in Secondary Schools ................................................ 27 3.1.2. Steedman (1983): Examination Results in Selective and Nonselective Schools........................................................................................................................... 30 3.1.3. Marks, Cox and Pomian-Srzednicki (1983): Standards in English Schools: Report No 1................................................................................................................... 32 3.1.4. Marks and Pomian-Srzednicki (1985): Standards in English Schools: Report No 2................................................................................................................... 40 3.2. More recent studies ................................................................................................................43 3.2.1. House of Lords, 2000.................................................................................................... 43 3.2.2. Jesson (2000): The comparative eva luation of GCSE value added performance by type of school and LA................................................................... 45 3.2.3. Jesson (2001): Selective systems of education – blueprint for lower standards? .................................................................................................................... 55 3.2.4. Jesson (2007): A ladder of opportunity? The pupil intake and performance of England’s grammar schools.................................................................................. 58 3.2.5. Prais (2001): Grammar School’s Achievements and the DfEE’s Measure of Va lue-added: an attempt at clarif ication........................................................... 59 vii EFFECTS OF SELECTIVE EDUCATIONAL SYSTEMS 3.2.6. Yang and Woodhouse (2001): Progress from GCSE to A and AS Level: institutional and gender differences, and trends over time. ............................ 60 3.2.7. Schagen and Schagen (2003): Analysis of National Va lue-added Datasets to Assess the Impact of Selection on Pupil Performance.................. 62 3.2.8. Schagen and Schagen (2005): Combining multilevel ana lysis with national value-added data sets – a case study to explore the effects of school diversity. ......................................................................................................... 66 3.2.9. Atk inson, Gregg and McConnell (2004): The result of 11 plus selection; An Investigation into Equity and Efficiency of Outcomes for Pupils in Selective LEAs. ........................................................................................................... 68 3.2.10. Burgess, McConnell, Propper, Wilson (2004). Sorting and Choice in English Secondary Schools....................................................................................... 73 3.2.11. Burgess, S., Propper, C. & Wilson, D. (2005): Will More Choice Improve Outcomes in Education and Health Care? The Evidence from Economic Research. The Centre for Market and Public Organisation ............................. 75 3.2.12. West, A. & Hind, A. (2006): Selectivity, admissions and intakes to ‘comprehensive’ schools in London......................................................................... 76 3.2.13. Levačić and Marsh (2007): Secondary modern schools: are their pupils disadvantaged? .......................................................................................................... 78 3.2.14. Maurin, E. & McNally, S. (2007): Educational Effects of Widening Access to the Academic Track: A Natural Experiment. ................................................. 81 3.2.15. Clark (2007): Selection versus Comprehensives: Wh ich Delivers the Best Educational Outcomes? .................................................................................... 84 3.3. Updates on the NCDS data...................................................................................................85 3.3.1. Sullivan and Heath (2002): State and Private Schools in England and Wa les............................................................................................................................. 85 3.3.2. Galindo-Rueda and Vignoles (2004): The Heterogeneous Effect of Selection in Secondary Schools: Understanding the Changing Role of Ability........................................................................................................................... 88 3.3.3. Manning and Pischke 2004: Ability Tracking and Student Performance in Secondary Schools in England and Wales ............................................................ 92 3.3.4. Manning, A. & Pischke, J. (2006). Comprehensive versus Selective Schooling in England and Wa les: Wh a t Do We Know? ................................... 93 3.4. Other countries.......................................................................................................................95 3.4.1. Research from Northern Ireland............................................................................. 95 3.4.2. Croxford (2000): Inequality in Atta inment at 16: A ‘Home International’ Comparison................................................................................................................... 97 3.4.3. Croxford & Paterson (2006): Trends in socia l class segregation between schools in England, Wa les and Scotland since 1984 ........................................... 99 3.4.4. Marsh (1991): The failure of high-ability h igh schools to deliver academic benefits: the importance of academic self-concept and educational aspirations .......................................................................................... 101 3.4.5. Hanushek & Wößmann (2006): Does educational tracking affect performance and inequality? Differences-in-differences evidence across countries ...................................................................................................................... 102 3.4.6. Wa ldinger, F. (2007): Does Ability Tracking Exacerbate the Role of Family Background for Students’ Test Scores? .................................................. 103 3.4.7. Jenkins, Micklewright and Schnepf (2006): Socia l segregation in secondary schools: how does England compare with other countries? ........ 105 4. Limitations of attempts to evaluate selective systems .........................107 4.1. Inability to control for other differences .............................................................................107 4.1.1. K...
View Full Document