Med Decis Making-2000-Chapman-451-8

8/7/2019 Med Decis Making-2000-Chapman-451-8

1/8

451

Health Economics

A Comprehensive League Table ofCost-Utility Ratios and a Sub-table of

"Panel-worthy" StudiesRICHARD H. CHAPMAN, MS, PATRICIA W. STONE, RN, MPH, PhD,EILEEN A. SANDBERG, MS, MBA, CHAIM BELL, MD,PETER J. NEUMANN, ScD

Objectives. The authors compiled a comprehensive league table of cost/QALY ratios,and a standardized table of analyses satisfying selected Reference Case criteria fromthe USPHS Panel on Cost-Effectiveness in Health and Medicine. Methods. Theyidentified 228 cost-utility analyses (CUAs) through literature searches, and abstracted

data on methods and cost-utility ratios. The subset of "Panel-worthy" analyses used:a societal or broad health-care perspective, community or patient preference weights,net costs, incremental comparisons, and discounting of costs and QALYs. Results.The 228 CUAs included ratios for 647 interventions, ranging from cost-saving to$52,000,000/QALY (median = $12,000/QALY). The standardized table presents 112ratios that met the "Panel-worthy" criteria, with articles published in recent years more

likely to meet all of the criteria. Conclusions. The comprehensive league table (avail-able on the Web) provides a useful reference, but ratios may not be comparable be-cause of methodologic variations. The standardized table focuses on studies meetingbasic methodologic criteria, potentially allowing for better comparison with future Ref-erence Case analyses. Future studies should investigate the quality of analyses un-derlying assumptions in addition to whether certain key procedural protocols were met.

Key words: cost-utility analysis; league tables; economic evaluation. (Med DecisMaking 2000;20:451-467)

Address correspondence and reprint requests to Dr. Neu-

mann. Harvard Center for Risk Analysis, 718 Huntington Avenue,Boston, MA 02115, telephone (617) 432-1312; fax. (617) 432-0190;

e-mail. &pr;[email protected] edu).

Ranked listings of the cost-effectiveness ratios ofvarious health and medical interventions, often

called &dquo;league tables,&dquo; have been used to facilitatecomparisons across cost-effectiveness analyses.Comparisons of ratios are essential because we do

not know whether a program is a good value untilit is compared with the benefits derived from re-sources expended in other programs. These leaguetables (so called after the tables used for British soc-

cer league standings) have often been criticized,however.1-4 The major criticism is that listing anal-yses in one table &dquo;implies a certain degree of ho-mogeneity in study methodology,&dquo; a homogeneitythat is usually missing.2 Drummond and colleagueslist methodologic features that are particularly im-

portant to consider when interpreting rankingswithin league tables, including: the discount rate,utility valuation methods, the included range ofcosts and effects, and the choice of comparison pro-gram. Mason proposed a similar &dquo;minimum data-set&dquo; that he recommended should be included with

each entry in a cost-effectiveness league table.3 Toreduce the effect of variability in source-study meth-

ods, these authors have recommended that leaguetables include only studies that are standardized onkey analytic practices.While complete standardization of league-table

entries is the ideal, most published analyses do not

allow this. A recent systematic audit of 228 cost-

Received October 22, 1999, from the Program on the Eco-nomic Evaluation of Medical Technology, Harvard Center forRisk Analysis, Harvard School of Public Health, Boston, Massa-chusetts (RHC, PWS, EAS, CB, PJN), the University of RochesterSchool of Nursing, Community and Preventive Medicine, Roch-

ester,New

York (PWS); the Institute of Medical Science, Univer-sity of Toronto, Toronto, Ontario, Canada (CB), and the Depart-ment of Health Policy and Management, Harvard School ofPublic Health (PJN) Revision accepted for publication March 22,2000 Presented in part at the annual meeting of the Society forMedical Decision Making, October 4, 1999, in Reno, Nevada. Sup-ported entirely by a grant from the National Science Foundationand Merck & Co., Inc. under the joint NSF/Private Sector Re-

search Opportunity Initiative (SBR-9730448). Mr Chapman was

supported by a training grant from the Agency for Health Care

Policy and Research, and Dr. Bell held a fellowship from theMedical Research Council of Canada, at the time of this study


2/8

452

Table 1 * Information Collected about Each Article

utility analyses (CUAs) published from 1976 to 1997

revealed wide variations in methods for estimatingand reporting costs, effectiveness, and preferenceweights, and little evidence of improvement overtime.5,6 This confirmed earlier findings of variationsin methods for estimating QALYS in CUAs.7Othershave reported that many published cost-effective-ness analyses do not adhere to commonly acceptedstandards.1,8-15 Furthermore, inadequacies of re-

porting and the space constraints of journal publi-cation often mean that the methodologic details ofCUAs are not disclosed.5

5

The purpose of this study was to use data from a

comprehensive review of published CUAs to createtwo league tables: 1) a comprehensive listing of in-

terventions and their cost/QALY ratios, and 2) a stan-

dardized sub-table that includes only ratios fromanalyses that adhere to key methodologic recom-mendations of the U.S. Public Health Service

(USPHS) Panel on Cost-Effectiveness in Health and

Medicine. 16,17 The Panel has set out guidelines forthe &dquo;Reference Case&dquo; analysis-an analysis usingstandard methodologic practices, intended to in-form resource-allocation decisions across studies.

Methods

Details of the study design, data collection, andanalytic plan are presented below.


3/8

453

SOURCE STUDIES AND DATA COLLECTION

We conducted a computerized literature search

through 1997 of the following databases: Medline,HealthSTAR, CancerLit, all editions of Current Con-

tents, and EconLit. We searched for the Medical

Subject Headings and/or text key words: &dquo;quality-adjusted,&dquo; &dquo;QALY,&dquo; and &dquo;cost-utility,&dquo; and identified

approximately 1,500 candidates for the database. Wethen screened the titles and abstracts to remove ob-

viously non-eligible articles, such as methodologic,review, and non-CUA studies. A final list of 228 orig-inal CUAs published in English from 1976 through1997 was compiled. A data-collection (or audit) formwas developed, and several versions of the formwere pilot tested before being put into use. The finalversion of the form had 69 fields on the reportingand methods used (table 1), including: study per-

spective ; themeasurement

of effectiveness, health-state preferences, and costs; the use of discounting;and whether incremental analyses were performed.We used the forms to abstract data from the 228

CUAs, with two readers per article. The data were

abstracted independently; then the two readers metto reach consensus on discordant items. The ab-

stracted information was recorded in a relational

database (FileMaker Pro, ver. 3.0). Further details of

the data-collection and auditing methods have been

reported elsewhere.s6

The baseline cost-per-QALY ratios from each ar-

ticle were also recorded in the database. Because

CUAs often compare several possible programs or

intensity levels, a single article could contributemore than one baseline CU ratio (with one to 18

ratios per article in this database). For example,while 45% of the studies in the database contained

one ratio each, 5% contributed ten or more ratios.

Ratios were reported as costs per QALY gained, cost-

saving (if the intervention was less costly but pro-duced more QALYs than its comparator), or domi-nated (if more costly and less effective). It is also

possible for the evaluated intervention to be bothless

costlyand less effective than the

comparatorprogram, yielding a CU ratio with negative numbersin both the numerator and the denominator. This

was true for two of the evaluated interventions in

our database (neither of which is in the &dquo;Panel-

worthy&dquo; subset of articles). Although a single (posi-tive) ratio can be reported in these cases, the inter-

pretation of these ratios is not the same as when theintervention is more costly but more effective. Re-

placing the comparator (status quo) program withthis type of intervention would yield a certainamount of savings per QALY foregone, so a higherratio is

actuallyconsidered more cost-effective than

a lower ratio.18 Because of this different interpreta-tion, we have added a footnote to the comprehen-

sive league table, indicating the two cases where thisoccurs.

For each cost-per-QALY ratio, we recorded a brief

description of the evaluated health-care interven-tion, a description of the baseline comparator, andthe target population for which the intervention is

intended (following a practice begun byGraham andVaupel19). We also classified each intervention intothe appropriate disease category (based on the ICD-9 classification system), intervention type, and levelof prevention (primary, secondary, or treatment).

ADJUSTMENTS TO PUBLISHED

COST-EFFECTIVENESS RATIOS

The CUAs in this database had been conducted in

several different countries and in different years. To

allow comparisons across countries, all non-U.S.

currency figures were converted into U.S. dollars,using the foreign exchange factor appropriate tothat country for the base year.2o In addition, the dol-lar values reported in these studies ranged from1976 to 1996. Each years nominal dollar amountswere inflated to real 1998 dollars using the generalConsumer Price Index (CPI).21 (Which version of theCPI to use in adjusting for health care inflation hasbeen the subject of some debate.2z z3 Some have ar-

gued for using the general rather than the medical

CPI, because of concern over whether the medical

component of the CPI adequately captures changesin the quality or intensity of medical services overtime.) Finally, all CU ratios were rounded off to two-digit precision.

DATA ANALYSIS

A comprehensive league table (published on theWeb at (www.hcra.harvard.edu/medical.html)) pre-sents the cost-per-QALY ratios as reported in the lit-

erature, with each studys costs inflated to a base

year. We then present several descriptive summarytables based on the information

gatheredin the da-

tabase. These tables summarize the studies in the

database, and stratify the information by the type ofintervention, prevention stage, and disease category.The methods used in each analysis were also

compared with key recommendations in the recent

report of the USPHS Panel on Cost-Effectiveness in

Health and Medicine} 1617 to determine the extent towhich the studies adhered to standard practices andwhether there is any evidence that methods have

been improving over time.A second league table (presented in the appendix

and also posted on the Web at (www.hcra.harvard.edu/medical.html)) includes only ratios from thosestudies that conformed to selected Panel recom-


4/8

454

mendations for estimating cost-effectiveness ratios.This table is designed to address criticisms thatcost-effectiveness ratios from different studies are

not comparable because methods may differ widely.

SELECTION CRITERIA FOR STANDARDIZED

LEAGUE TABLE

We used the following Reference Case recommen-dations to identify the subset of data points to in-clude in the standardized league table: adoption ofa societal perspective, community or patient pref-erence weights, use of net costs, appropriate incre-mental comparisons, and discounting of costs and

QALYs at the same rate. These recommendations

are summarized below.

Societal perspective. Reference Case analyses areintended to aid societal resource allocation deci-

sions. The Panel noted that to allocate societal re-

sources efficiently, all relevant costs and benefitsshould be included, and therefore Reference Case

analyses should adopt a societal perspective. Be-cause only about half of the studies in the database

explicitly stated the adopted perspective, we usedour own judgment of each studys perspective. Also,because relatively few studies (53, or 23%) used atrue societal perspective (e.g., including non-healthcosts such as time or travel costs), we relaxed this

criterion to include data points from any study thatwe judged took a societal or broad health-care sec-tor perspective (i.e., inclusion of all health-care costs

regardless of who incurred them).Preference weights from community or patients.

The Panel argued that community preferences (that

is, those of the general population) are best for Ref-erence Cases, because they are more likely to rep-resent societal preferences for resource allocation.However, the Panel also noted that patient prefer-ences for health states would be acceptable in a Ref-erence Case analysis, if community preferences arenot available and are

prohibitively difficult tomea-

sure. We therefore include studies that used either

the general population or patients as the source ofhealth-state preferences (thus excluding studies thatrelied on clinicians or authors judgments of pref-erence weights).

Use of net costs. Health care interventions not onlyincur short-term costs or benefits, but also may off-set downstream costs by preventing long-term mor-

bidity. Reference Case analyses should therefore cal-culate costs net of any such downstream savings.The Panel also recommended the inclusion of direct

health care costs as well as direct non-health care

and time costs. While almost all of the studies in this

database (99%) included some direct health care

costs, fewer (17%) included any non-health care or

time costs.6 We included only analyses that calcu-lated costs net of any downstream savings, as rec-ommended by the Panel, but did not screen studiesfor the types of costs included.

Appropriate incremental comparisons. Some stud-ies reported an average cost-effectiveness ratio fora program, rather than incremental ratios between

programs. An average cost-effectiveness ratio is one

calculated in comparison with a hypothetical no-cost, no-life-expectancy baseline rather than in com-parison with the next-most-effective program.16(P

285)

For studies that reported sufficient data, we con-verted average CU ratios to incremental ratios

FIGURE 1. Distribution of estimated cost-utilityratios


5/8

455

Table 2 * Median Cost-Utility Ratios by Type of Intervention

by identifying the appropriate comparison programand recalculating the cost-per-QALY ratio. The stan-dardized league table includes only analyses withappropriate incremental CU ratios.

Discounting of costs and Q~ILYs. The Panel rec-ommends discounting of both costs and health con-

sequences at the same rate (3%). They also recom-mend sensitivity analyses with a discount rate of 5%,to allow comparison with previous studies. For thestandardized table, we included data points from

any article that discounted costs and health effects

at the same rate, either 3% or 5%, or for which the

time horizon was one year or less, so that discount-

ing was not necessary.

Results

The 228 CUAs included data for 647 intervention-

comparator pairs, which are collected in a compre-hensive league table, along with associated cost-effectiveness ratios in 1998 dollars per QALY. The

comprehensive league table is not shown here, butis available on our Web site, (www.hsph.harvard.edu/organizations/hcra/hcra.html). The interven-tions are grouped by disease category and by typeof intervention within disease categories. Sixty-nine(10.6%) of the interventions were reported to be cost-

saving and 64 (9.9%), dominated. The distribution ofestimated CU ratios ranged from cost-saving to

$52,000,000/QALY, with an overall median (excludingdominated interventions) of $12,000/QALY (figure 1).The distribution is positively skewed, with 69% of the647 interventions estimated to have CU ratios of

$50,000 or less, and 78%, $100,000 or less.When categorized into different types of interven-

tions, the median CU ratios varied considerably (ta-ble 2), ranging from a median of $2,000/QALY for

immunizations to $140,000/QALY for &dquo;other medical

procedures&dquo; (again excluding dominated interven-

Prevention level

FIGURE 2 Cost-utility ratios by interventions level ofprevention(outliers excluded) Thick lines indicate medians, and boxes the

interquartile ranges.

tions). However, the median CU ratios were under

$70,000/QALY for all intervention types except those

classified as other medical procedures. The other-

medical-procedures category includes some inter-ventions that have the highest estimated CU ratiosin our database, such as preoperative autologousblood donations. The numbers of ratios recorded in

the database also varied greatly by intervention type(table 2), with the largest categories being pharma-ceutical and surgical interventions (with 233 and 119ratios, respectively).The interventions were also classified according

to level of prevention: primary, secondary, or terti-

ary (that is, treatment). Primary prevention includedmeasures intended to prevent the onset of targetedconditions; secondary prevention was defined asmeasures aimed at identifying and treating asymp-tomatic persons who have already developed risk


6/8

456

Table 3 * Median Cost-Utility Ratios by ICD-9 Disease Category

*Includes intenrentions that were not aimed at specific diseases, or included more than one diseasecategory.fThe congenital anomalies category included one non-dominated intervention, estimated to be cost-saving; the pregnancy, childbirth, and puerpenum

category included one intervention, with an estimated ratio of $1,500/QALY.

factors or pre-clinical disease.24 Tertiary preventionrefers to treatment for clinical diseases or condi-

tions. As shown in figure 2, the median CU ratiosare comparable for health-care interventions aimedat primary secondary, and tertiary prevention (at$14,000, $14,000, and $11,000 per QALY, respec-

tively). However, the ratios for primary preventioninterventions were much more varied than those for

secondary or tertiary prevention (with an interquar-tile range of $407,000 for primary vs $34,000 for sec-

ondary and $39,000 for tertiary prevention).The numbers of analyzed interventions varied

considerably by disease category (table 3), with some

relatively well studied, such as circulatory system,infectious and parasitic, and neoplasms. Other dis-ease categories, such as congenital anomalies, injuryand poisoning, and pregnancy, childbirth and pu-

Table 4 * Numbers of interventions and Articles MeetingSelected Reference Case Criteria

erperium, had ten or fewer interventions for whichCUAs had been conducted. The median CU ratios

ranged from cost-saving for interventions targetedat diseases of the blood and blood-forming organsto $40,500/QALY for interventions in respiratory sys-tem diseases (table 3).

Applying our selected Reference Case criteria, wefound that CUAs for 95% of the interventions were

judged to take a societal or broad health-care sectorperspective (table 4). Analysts used the general com-

munity or patients as the source of preferenceweights for 39% of the intervention-comparatorpairs. Net costs (defined as costs net of any down-stream savings) were calculated for 83% of the ratios.

Appropriate incremental comparisons had been

performed for 49% of the ratios. Analysts appropri-ately discounted both costs and QALYs for 80% ofthe CU ratios. Overall, 112 of the 647 interventions(or 37 of the 228 articles) used all of these recom-

mended basic analytic methods, representing 17% of

the ratios in the database (table 4; appendix). Thenumber of interventions meeting the selected Ref-erence Case criteria over time is shown in figure 3.A higher proportion of the interventions publishedin recent years than those published before 1991were in the &dquo;Panel-worthy&dquo; subset (figure 3). These&dquo;Panel-worthy&dquo; interventions and their associatedratios are listed in the appendix to this article.

Discussion

The comprehensive league table includes 647cost-per-QALY ratios that vary across a range from


7/8

457

cost-saving to $52,000,000/QALY to dominated. Thisvariation is related not only to the intervention beingstudied, but also to the type of intervention, disease

category, and prevention level. While we found that

only 17% of the ratios in our database adhered tothe selected USPHS Panel guidelines, figure 3 pro-

videssome

evidence that,over

time, analysts may beconverging toward a consensus on methodologicpractices, a trend that should only be accelerated bythe publication of the Panels guidelines in 1996.This study adds to previous work in the field by

concentrating on studies that report costs per QALY,and by incorporating the Panels recommendationsfor the conduct of cost-effectiveness analyses. Whilethere have been comprehensive bibliographies ofeconomic evaluations in genera 121,11 (and extensive

league tables of cost-effectiveness analyses usingcosts per life year as their metric27)} this article fo-cuses

onlyon

analyses that report costs per QALY,as the metric recommended by the USPHS Panel on

Cost-Effectiveness in Health and Medicine. The

comprehensive, Web-based CUA database can helpdecision makers to make more meaningful compar-isons of diverse medical interventions. By consultingthe original published analyses, users can assess the

comparability, relevance, and reliability of the re-ported CU ratios. But while analyses of the compre-hensive CU ratio list provide a useful reference ofthe field up until 1998, they also illustrate the chal-

lenge of comparing ratios given wide methodologic

variations. Mason3has classified critiques of cost-per-QALY league tables into two main types: one fo-cused on the appropriate way to present league-table information, and the other on theoretical

problems with using league tables to improve allo-cative efficiency. This article addresses the formercriticism by highlighting ratios from those analysesthat used uniform, recommended methods for con-

ducting CUAs. The &dquo;Panel-worthy&dquo; subset of CU ra-tios may serve as a better benchmark for compari-sons with new analyses, especially those intendedfor use as reference cases.

There are several limitations to the analyses andleague table presented here. While our literaturesearch included several computerized databases, wedid not identify CUAs that were either available innon-listed publications or not published at all. Be-cause we are limited to published studies, any pub-lication bias in CUAs would be reflected in the da-

tabase, as well as what might be called study bias,in that certain diseases and intervention types are

relatively well studied (for example, pharmaceuti-cals) compared with others. In addition, because wenarrowed our search to certain key words (&dquo;quality-adjusted,&dquo; &dquo;QALY,&dquo; or &dquo;cost-utility&dquo;), we may havemissed some published studies that estimated costs

Year of publication (grouped)

FIGURE 3. Numbers of interventions, grouped by year of publi-cation, and numbers meeting selected Reference Case criteria

(Pearson chi-square = 40.2 with 4 degrees of freedom, p

Med Decis Making-2000-Chapman-451-8

Documents

Transcript of Med Decis Making-2000-Chapman-451-8