use http://www.jstor.org/stable/3084598
Your of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=aaas. Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with the scholarly community to preserve their work and the materials they rely upon, and to build a common research platform that promotes the discovery and use of these resources. For more information about JSTOR, please contact [e-mail].
American Association for the Advancement of Science is collaborating with JSTOR to digitize, preserve and extend access to Science.
http://www.jstor.org
3
POLICY FORUM: GLOBAL HEALTH
a _ le
_
WHO
Ranking
of
Health
System
For
Performance
meaningfully be constructed, and how, from
Upperbounc
Dean T.Jamison* and Martin E.Sandbu
dEbates!~~PXllbl
,.. ,,
Health Report 2000 ' org/cgi/content/summary/ t 293/5535/1595 (WHR2000), pub-
t overhalf a century, he availability of economic performance indicators-such as GDP per capitaand inflationrates-has made it possibleto hold f politicalleadersaccountableor economic management. Equally important, these economicoutcomemeasures(and the entiresystemof national ncomeandproduct i accounts)have allowed evidence to supplant ideology for judging the soundness m of alternative acroeconomic olicies. p a Publication f robust,transparent, nd o validindicesof healthsystemperformance could, likewise, lead to greaterpolitical accountability and to evidence-based health policies. To this end the World HealthOrganization Respondonline (WHO),in its World http://www.sciencemag.
Only 20% of Italiansratetheirhealthcare a I systemas satisfactorylthoughtalyis number 2 in the WHR2000ranking. enmark, D h ranked16 out of the 17 by WHR2000, ad 91%of its citizensconveysatisfaction6). ( b d Giventhe dramatic ifferences etween t rankings,it is important o examinehow they are derived(7, 8). The figure on this t page illustrates he WHR2000methodology. The heavy line in
ODm;
-
the middle traces out the health outcome that
d available ata. / For many purposes, ' O1' H outcomelevelswill con/ODjvey performance.That < HO1 P Canada'smale life ex- o / pectancy in 1999 of 76.2 yearsexceededthat of the United States by / / "Lower bound 2.4 yearsprovidesvaluW able information. hen d countries iffermarked~i\~ ~ ly in income, however, r Health e ources (HR) relative levels may
would be statistically predicted (PHO) for a countrygiven its health expenditures. he actuT al health outcome for countryi will typically deviate from the predicted outcome, and
this is illustrated by the
distance HOi - PHOi is more informative. Measuring erforna ce of health sys- the outcome deviation prove n p The term "country er- tems inWHR2000 heheavyineplots for countryi, ODi. p l L.1 r
o placement f HOi(actual health outcome in countryi). The vertical
~ lished indices of health system performance for its 191 member states (1, 2). WHO's farsighted h leadership as initiateda processthatwill ultimatelyimprovethe evidence base for health policy. We argue, however, that h WHO'scurrent erformance lgorithm as a p criticalshortcomingsndthatthe challenge a r v of constructingalidmeasures emains.
Performance Measures In its World ealthReport1999 (WHR1999), H
m WHOpublished easures y whichcountry b t r could performance be ranked elativeo what would be predictedby income level (3). Rankings of health system performance would add substantially o knowledgeof t f overallcountry erformanceor evaluating p and improving health policy. In its W WHR2000, HOseeks to disentangleyss from of temperformance otherdeterminants T c health utcomes. heresulting ankings oro r relateonly slightlywith the 1999 country t rankings-raisinghe questions performance m c of whattypesof performance easures an
D.T.Jamisonis with the Program n GlobalHealth o o and Education,choolof Public ealth,University f S H L C California, osAngeles, A90095, USA.M.E.Sandbu T is in the Centerfor International evelopment, he D S H U Kennedy choolof Government, arvard niversity, M Cambridge, A02138, USA. *To whom correspondenceshould be addressed. E-mail: [e-mail] d
formance" indicates a statistical redic on of healthoutIn WHR2000, the i p cti how well a country is comeas a functior fhealth esources.authors define perforIo r b and l doing relative to what Abbreviations th e relation etween mance in terms of an f wouldbe predictedrom the prediction aindthe bounds re upperboundthatwould a line o be achievedby a maxiin ii incomeandperhaps th- explainedthemaintext(13). er specified determimally efficient health i nants(4). Country erformancendicators systemanda lowerboundthatis "theleast p providea startingpoint for discussionof that could be demanded..."[p. 41 in (1)]. m why a country aybe doingwell orpoorly. "Maximalefficiency"was assumedto be o l That a country's erformances favor- the prediction ine plus the maximum uti p able could result from multiple factors: come deviation, for any country,ODmax. h high levels of healthexpenditure,igh effi- The lowerboundsomehowemergesfrom d ( ciency in use of healthexpendituresgood early20th-century ataon today'shigh-inf healthsystemperformance),avorable e- come countries. g o g ography, ood governance, r luck. To asSpecifically,the WHR2000index india sess healthsystemperformance,s opposed cates how far a countryis abovethe lower to countryperformance,equiresidentify- bound (LBi) as a fractionof the distance r r ing how outcomeforeachcountry esponds betweenthe upperandlowerbounds: W to a changein inputs. HR2000 implyass v sumes that system performance ariation country performance = [OD, + (PHO, accounts for all outcome variationafter LBi)]/[ODmax + (PHO,- LB,)] f controlling or levels of healthexpenditure c and education.No outcome variationre- In the two-dimensionalontextof the figo sults from otherdeterminantsf healthor ure, the WHR2000methodcan be viewed o i m c fromlimitationsn theunderlying odel. as a country-specific orrection f the upt b A As mightbe expected,he correlatione- per boundto adjustfor education. s with h m a tweenWHR2000 easures ndcountry er- countryperformance, ealth system perp formances low:Twenty utof 96 countries formancebecomes a functionof ODi, ali o one. movedeitherup or downby 25 percentile beit a complexandnontransparent Threepoints are clear:Includingaddipoints or more betweenthe two rankings. T w 1 Forexample, heGambia's 999placement tional determinants ill explain more of w at the 9th percentile ontrasts ith its 2000 the outcome deviation, but by different c ranking at the 50th (5). Likewise the amountsin differentcountries.Hence,the t WHR2000 rankingcorrelatesnegatively relativesizes of the residualsattributedo with citizens' satisfactionwith their own healthsystemefficiencycouldbe markedhealth care system for 17 high-income ly affected.Second,even with all plausible o for o countries f the Organization Economic controlsincluded,shortcomings f model and Development(OECD). anddataimplythatremaining utcomedeo Cooperation 1595
2001 S V2 www.sciencemag.org CIENCE OL 93 31 AUGUST
SCIENCE'S
COMPASS
viation will result only partly from effiT ciency variation. hird,multiplearbitrary assumptionsdefine the upper and lower s bounds,leavingrankings ensitiveto these assumptions. Analysis Sensitivity How quantitatively a importantretheseconcers? We assessedthis with a sensitivity analysisby adjustingthe upperboundfor v i.e., geography, we addedgeographicalaria ablesto healthexpenditures nd education in predicting utcomes. Tropicalocations, o l ( forexample, ppearo affecthealthadverset a t ly.) We correspondinglydjusted he upper a t bound, henrecalculatedankings9, 10). ( r
0. 10c.
c ings if g, ographical ontrolswere added. e Thisanal sis onlypointsto greatsensitivity y in results henvariablesin additiono eduw t ( r T cation)ai econtrolled. he conceptual robp lem is more fundamental.ome of a counS c try's out<ome deviationresults from how I well its healthsystemperforms.t could,for a particul country, e 10%of the outcome lar b deviationit couldbe 50%,it couldbe any, I thing. W HR2000 assumes 100% for all countriesbecauseit lacksa wayof estimat, t c ing the a( tualvalues.We also examinedhe t sensitivit of the rankings o different ery p a centageassumptions, nd as with the findi ings on geography, the rankings differ r. markedlyThe authorsof WHR2000offer no empirical justification for theirassumptionf 100%. o
/
of a counterproductiveffect if technical e a mistakesremainuncorrected nd resultant rankingsunsupportable.Directions suggested in this note will, we hope, contributeto the evolutionof WHO'simporinitiative. tantperformance easurement m
References ndNotes a
1. WHO, WorldHealth Report2000-Health Systems: ImprovingPerformance(WorldHealth Organization, Geneva,2000). 2. WHR2000 provides rankingsof health systems with respect both to health (disability-adjusted life exa pectancies or DALEs) nd overall performance.Overall performance combines measures of attainment not only on health but also on aspects of finance,reT sponsiveness, and distribution. his policy forum discusses performancewith respect to health, but the same issues arise concerningoverallperformance.For a critical discussion of the content and construction of all these measures see A. Williams, Health Econ. 10, 93 (2001). 3. WHO, World Health Report 1999: Makinga Difference (WorldHealth Organization, eneva, 1999). Its G Annex table 6 reports country performancerelative to income between 1952 and 1992 on infant mortality rate and female life expectancy. One author of this policy forum (D.T.J.)was the lead author of WHR1999. 4. Similar analysis is possible at the level of hospitals rather than countries. California, for example, assessed performance differences in management of acute myocardialinfarctionrelative to variablesthat control for the complexity of the case mix presenting at the hospital. [See H. S. Luft,P.S. Romano,Principal Investigators,Second Reportof the California ospiH tal Outcomes Project,vols. 1 and 2 (California ffice O of Statewide Health Planning and Development, differed markedly in risk-adjusted 1996). Hospitals performance. 5. The percentiles referto the countries'WHR1999and WHR2000 ranks,respectively,relative to the subset of countries included in both rankings.WHR1999 used level of female life expectancy in 1992 as its outcome measure, whereas WHR2000 used DALEn i 1997. Data on life expectancy and DALE re virtually a perfectly correlated, so use of DALEs mposes the i cost of lower transparencyfor virtually no gain, and in particular,this is not a source of difference between the WHR2000 and the WHR1999 rankings. (Female life expectancy in 1992 has a 0.95 correlation with DALEsn 1997.) i 6. See R.J. Blendon,M. Kim,J. M. Benson, Health Affairs 20,10 (2001). 7. The WHR2000 algorithm was proposed by C. J. L. a B Murray nd J. Frenk, ull.W.H.O.78(6), 717 (2000). C 8. The method is describedin D. Evans, .Tandon, . J. L. A A. Lauer, Thecomparativeefficiency of na" J Murray, . tional health systems in producinghealth:An analysis of 191 countries" (Global Programon Evidence discussion paper no. 29, World Health Organization, Geneva,2000). 9. Because the actual frontier values for each country were not publishedby WHO,we approximatedusing informationin D. B. Evanset al. (8). We regressedthe measure of health outcome on the same variables they used plus two geographicalvariables provided by J. L.Gallupand J. Sachs, BrookingsPap. Econ.Activity 2, 207 (1998). 10. D. Bloom and J. Sachs (9) find that tropical latitudes and geographicalisolation adverselyaffect countries' growth prospects because of less-productiveagriculture, a more hospitable environment for major diseases, and more difficultyin intellectualcontact with the rest of the world. 11. WHR2000 includes exceptionally precise estimates of the ranksof Tanzaniaand Uganda:Both countries were reported to lie within an uncertainty interval includingonly ranks179 and 180. 12. Fora discussion of these methods, see A. S. Brykand S. W. Raudenbush,HierarchicalLinearModels (Sage C Publications, ewburyPark, A,1992). N 13. The first and second figure are based on figures 2 and of (8). 9, respectively, 14. The authors appreciate valuable input from G. Alleyne, E. Bos, W. Hsiao, P. Jha, P. Musgrove, S. Nguyen, X. Pitkow, J. Sachs, and J. Wang.
A Empirical ssessment of Performance m 20Is it possible to estimate the 0 amountof a country'soutcome ,._ 303 0*..-Kuwait deviation that is, indeed, at40tributableto how efficiently it uses resources? imeseriesdata T 50*.Bolivia i on healthexpenditures,n addi' ? . tion to the other relevantvari1. 60t ables,wouldallowan attempt o estimatestatisticallyhe respont siveness(or elasticity)of health outcomeswith respectto health f expenditureseparatelyor each s " 90country. If such an approach 100 i succeeded, t wouldprovideone 100 90 80 70 60 50 40 30 20 10 0 t potentialempiricalapproach o Percentile rank in WHR2000 defining health system perforEffect of adding geographical variables to WHR2000 mance. Methods of multilevel rankings (13). The dots relate a country'sWHR2000per- modelingused in educationrecentile rankon healthoutcomesto the rank eneratedfrom search provide examples for g a sensitivity analysis that uses WHR2000'smethods but proceeding12). ( Eventhe moremodestobjeccontrolsfor geographicaleaturesof the country. ncertainf U W a c p ( ty intervals verticalbars)indicate HR2000 uthors' onfi- tive of assessingcountry erforf F dence in the rankingor each country. orexample,Bolivia's mance requires substantial caveats.That said, countryperrangewas fromthe 25th to 29th percentiles blackbar). ( formance measurementcould The figureon thispageplotseachcoun- be subs tantially improved over what 9 try's geography-adjusted ercentile rank WHR19 9 reported.In particular,meap t r c against he WHR2000 ank.Rankings oin- sures of countryperformancerelative to cide along the 45? line. Verticalbars display geograpihy,in addition o incomeand edut the "uncertainty interval"or eachcountry's cation, are probably lose to the best that c f ranking as given in WHR2000 (in per- can be dlonewithouttime series measures F d centiles). oronly17 outof 96 countries oes of healthresources. the geography-adjusted fall withinthe rank i interval. olivia, orexample, er- Conclusion f B uncertainty p r a t formed oorly ccordingo WHR2000, ank- In the pEst severalyears, the WorldBank p a A f [ ing at the 26thpercentile. fteradjustingor and WHO have published quantitative i B e h geography, owever, olivia'spercentilen- measure s of country performance in creases o 52, i.e.,by morethansix timesthe health. I y highlighting its rankings in t B widthof its uncertainty interval see figure, WHR20'0, and by attributing he results t 0 ( thispage).Thissuggests hatthe uncertainty to differ nces in the efficiency of health t e e intervals onveyno information r, worse, systems, WHO attracted xtensive media o c that heymisleadingly i. attention WHR2000has both stimulated convey recision11). ( t p We arenot claimingthatthe WHR2000 andcontributedo a much-needed ebate. t d r w algorithm ouldgenerate eaningful ankm High visibility runs the risk, however, 1596
31 AUGUST 001 2 VOL293
SCIENCE www.sciencemag.org