Can We Reliably Benchmark HTA Organizations? Michael Drummond Centre for Health Economics University of York

Transcription

1 Can We Reliably Benchmark HTA Organizations? Michael Drummond Centre for Health Economics University of York

2 Outline of Presentation Some background Methods Results Discussion

3 Some Background In recent years, several reimbursement agencies have begun using HTA. Essentially, HTA has become hard-wired into the reimbursement or coverage decision. Economic evaluation has an increasing prominence in these HTAs (for resource allocation decisions). It is important to establish some principles of good practice.

4 Methods (1) Developed 15 Key Principles for the Improved Conduct of HTA for Resource Allocation Decisions, based on the existing literature and our own views. Solicited comments from key opinion leaders in HTA. Published in IJTAHC 2008;24:244-58, along with editorial comments.

5 Methods (2) Selected 14 HTA entities from 9 countries, to reflect: - worldwide coverage; - traditional HTA agencies and reimbursement agencies; - established and newly-formed entities; - public and privately funded entities. Assessed whether the entities supported and/or implemented the various principles. Used our own expertise, supplemented by local experts for Australia, Brazil, Canada, Korea and Taiwan.

6 Methods(3) Based on the preliminary effort, several issues were raised A series of audit questions were developed which, it is argued, would enable us to benchmark HTA organizations in a more reliable manner

7 Results The Key Principles The Assessments Additional considerations in benchmarking HTA organizations Audit questions for the Key principles

8

9 Key Principles for HTA Structure of HTA Methods of HTA Processes for Conduct of HTA Use of HTA in Decision-making

10 Structure of HTA The goal and scope of the HTA should be explicit and relevant to its use. HTA should be an unbiased and transparent exercise. HTA should include all relevant technologies. A clear system for setting priorities for HTA should exist.

11 Methods of HTA HTA should incorporate appropriate methods for assessing costs and benefits. A full societal perspective should be considered when undertaking HTAs. HTAs should explicitly characterize uncertainty surrounding estimates. HTAs should consider and address issues of generalizability and transferability. HTAs should consider a wide range of evidence and outcomes.

12 Processes for Conduct of HTA Those conducting HTAs should actively engage all key stakeholder groups. Those undertaking HTAs should actively seek all available data. The implementation of HTA findings needs to be monitored.

13 Use of HTA in Decision-Making HTA should be timely. HTA findings need to be communicated appropriately to different decision-makers. The link between HTA findings and decision-making processes needs to be transparent and clearly defined.

14 Key Principles: How Widely Are They Adopted?

15 Assessment Criteria + Means that the organization supported the Principle in written guidelines or other forms, regardless of whether it was followed. ++ Means that the organization implemented the Principle in published reports and that the decisions based on these reports demonstrate adoption of the Principle.

16 Key Principle Structure of HTA program CMS (US) Washington Medicaid/ DERP (US) WellPoint (US) BCBS TEC (US) NICE (UK) IQWiG (Germany) DIMDI (Germany) 1 The goal and scope of the HTA should be explicit and relevant to its use 2 HTA should be an unbiased and transparent exercise 3 HTA should include all relevant technologies 4 A clear system for setting priorities for HTA should exist Methods of HTA HTA should incorporate appropriate methods for assessing costs and benefits 6 HTAs should consider a wide range of evidence and outcomes 7 A full societal perspective should be considered when undertaking HTAs 8 HTAs should explicitly characterize uncertainty surrounding estimates 9 HTAs should consider and address issues of generalizability and transferability

17 Key Principle Processes for conducting HTA 10 Those conducting HTAs should actively engage all key stakeholder groups 11 Those undertaking HTAs should actively seek all available data 12 The implementation of HTA findings needs to be monitored Use of HTA in decision making CMS (US) Washington Medicaid/ DERP (US) WellPoint (US) BCBS TEC (US) NICE (UK) IQWiG (Germany) DAHTA@ DIMDI (Germany) HTA should be timely HTA findings need to be communicated appropriately to different decision makers 15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined Notes: + signifies that the organization supported the principle in question in written guidelines or other form, regardless of whether they actually follow it. ++ means that the organization implemented the principle in published reports, and decisions based on these reports demonstrate adoption of the specific principle.

18 Key Principle TLV (Sweden) SBU (Sweden) CADTH (Canada) HIRA (Korea) PBAC (Australia) Anvisa (Brazil) DHTA (Taiwan) Structure of HTA program 1 The goal and scope of the HTA should be explicit and relevant to its use 2 HTA should be an unbiased and transparent exercise 3 HTA should include all relevant technologies 4 A clear system for setting priorities for HTA should exist Methods of HTA HTA should incorporate appropriate methods for assessing costs and benefits 6 HTAs should consider a wide range of evidence and outcomes 7 A full societal perspective should be considered when undertaking HTAs 8 HTAs should explicitly characterize uncertainty surrounding estimates 9 HTAs should consider and address issues of generalizability and transferability

19 Key Principle TLV (Sweden) SBU (Sweden) CADTH (Canada) HIRA (Korea) PBAC (Australia) Anvisa (Brazil) DHTA (Taiwan) Processes for conducting HTA 10 Those conducting HTAs should actively engage all key stakeholder groups Those undertaking HTAs should actively seek all available data 12 The implementation of HTA findings needs to be monitored Use of HTA in decision making 13 HTA should be timely HTA findings need to be communicated appropriately to different decision makers 15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined

20 Summary of Findings Support of principles High: HTA should be unbiased and transparent Low : HTA should adopt a societal perspective Implementation of principles High: HTA should consider a wide range of evidence Low: Link between HTA and decision-making should be clearly defined

21 Discussion Points Only a convenience sample of HTA entities. Different entities have different remits, which affects the adherence to some principles. The Key Principles merit more discussion: -some are controversial -some may be too idealistic -some are partially contradictory. Judgements of implementation are impressionistic and we need a tighter, audit approach.

22 Additional Considerations in Benchmarking Understanding the varying statutory roles of HTA organizations Weighting the relative importance of the Key Principles Recognizing the varying stages of development of HTA organizations Developing unambiguous audit criteria

23 Understanding the Varying Statutory Role of HTA organizations The remit of an organization can limit its ability to adhere to a principle Some principles (eg. KP 3 HTAs Should Consider All Relevant Technologies) relate to the whole jurisdiction, not an individual organization Therefore, need to distinguish between maximum score and maximum attainable score Nevertheless, some elements of the remit need to be questioned within the jurisdiction (eg KP 7 HTAs Should Adopt a Societal Perspective)

24 Weighting the Relative Importance of the Key Principles Clearly, all key principles may not be of equal importance The relative weightings could vary by jurisdiction There are trade-offs between principles (eg between KP 10 on Stakeholder Involvement and KP 13 on Timeliness) Trade-offs should be discussed at the level of the individual jurisdiction

25 Recognizing the Varying Stages of Development of HTA organizations It takes time to formulate a view about the various key principles As HTA organizations undertake studies/make decisions, case-law is developed Initially, the organization may not have the human resources to undertake all the tasks implied by the principles, even if it wanted to

26 Developing Unambiguous Audit Criteria Need audit criteria that can be defended The objective should be to develop a series of questions that can be answered yes or no, or graded in an unambiguous manner (eg a Likert Scale) Paper published in the International Journal of Technology Assessment in Health Care 2012;28(2):

27

28 Audit Criteria for the 15 Key Principles The objective was to create, from the 15 Key Principles, a series of questions that can unambiguously be answered This preliminary list is based on the points we raise in the text of the original Key Principles paper It may be possible to supplement this list with other questions

29 Principle1.The goal and scope of the HTA should be explicit and relevant to its use. Is the remit for the HTA organization clearly defined? Is a scoping document drawn up prior to an HTA, containing information on the specific decision problem to be addressed, the alternatives to be considered, the relevant patient populations and the time horizon?

30 Principle 2. HTA should be an unbiased and transparent exercise Is the HTA organization independent of the body making the reimbursement or coverage decision? Are the recommendations of the HTA organization made by an independent expert advisory committee? Are the meetings of the committee held in public? Are all supporting information and the basis of the recommendations made publicly available? Does the organization normally commission outside groups to undertake the HTA? Are the reports produced subjected to independent peerreview prior to final determination? Are the draft conclusions subject to review by stakeholders and the public, with rationale underlying final determinations of contentious issues?

31 Principle 3. HTA should include all relevant technologies Are all types of technologies (e.g. drugs, devices, diagnostics, procedures, behavioural modification) considered? Within each category, are both new and existing technologies considered? In assessments of new technologies, are all relevant alternatives considered?

32 Principle 4. A clear system for setting priorities for HTA should exist Does a formal system for prioritizing and selecting topics exist? Is the priority-setting approach clear and transparent?

33 Principle 5. HTA should incorporate appropriate methods for assessing costs and benefits Does the HTA organization consider costs as well as benefits and harms? Does the HTA organization have published methods guidelines for assessing the benefits, harms and costs of health technologies? Is a full systematic review of clinical evidence required as a basis for economic modeling? Does the team undertaking HTAs on behalf of the organization include individuals with skills in epidemiology/biostatistics, health services research and economics?

34 Principle 6. HTAs should consider a wide range of evidence and outcomes. Does the relevant clinical evidence include observational and non-randomized studies, as well as RCTs? Does the HTA consider impacts on quality of life and other patient-reported outcomes, as well as clinical events? Does the HTA consider relevant sub-groups of the patient population (e.g. by baseline risk)?

35 Principle 7. A full societal perspective should be considered when undertaking HTAs Does the HTA consider only consider the impact on a specific budget, for example for drugs? Does the HTA consider all health care costs? Can other costs be included as extra information? Are productivity gains and losses (ie indirect costs and benefits) considered when relevant? Are costs for informal care included when relevant? Are costs in added years of life included in the costeffectiveness ratio?

36 Principle 8. HTAs should explicitly characterize uncertainty surrounding estimates Does the HTA include a sensitivity analysis? Are confidence intervals presented for key estimates? Have the key deficiencies in available data been identified and discussed? Is an agenda for key future research proposed?

37 Principle 9. HTAs should consider and address issues of generalizability and transferability Does the HTA organization have methods guidance for dealing with transferability issues when using data or analyses from other jurisdictions? Does the HTA organization consider the generalizability of the results of its studies to other patient populations, health care delivery systems or practice settings that are relevant for its jurisdiction?

38 Principle 10. Those conducting HTAs should actively engage all key stakeholder groups (eg professional bodies, patient organizations, manufacturers). Is the HTA organization formally required to engage stakeholders in its activities? Does the HTA organization involve stakeholders in the scoping of HTAs? Does the HTA organization have a mechanism for identifying the relevant stakeholders Does the organization encourage or require submissions of evidence from stakeholders? Does the organization allow stakeholders to comment on reports at the draft stage? Does the organization allow stakeholders to appeal against recommendations/decisions? Do the organization s committees include stakeholder representation (eg patient groups, technology manufacturers, clinical specialists)?

39 Principle 11. Those undertaking HTAs should actively seek all available data Does the systematic review of clinical evidence include the grey literature and unpublished data? Does the HTA organization have processes for handling confidential data from manufacturers?

40 Principle 12. The implementation of HTA findings needs to be monitored Does the HTA organization develop an implementation plan for its HTAs? Does the HTA organization monitor the impact of its recommendations?

41 Principle 13. HTA should be timely. Does the HTA organization have a defined time period for conducting HTAs/producing recommendations? Does the HTA organization adhere to the agreed timelines? Does the organization have a mechanism to update its HTAs/recommendations within a given time period?

42 Principle14. HTA findings need to be communicated appropriately to different decision-makers. Does the HTA organization develop a communications plan for its recommendations and decisions? Are separate versions of reports produced for different audiences (e.g. health professionals, decision-makers, general public)? Is the effectiveness of communication monitored and evaluated?

43 Principle 15. The link between HTA findings and decision-making processes needs to be transparent and clearly defined Does the organization distinguish between the scientific assessment of the evidence and the appraisal decision? Does the organization have an explicit decision rule for acceptance/non-acceptance of health technologies? Does the organization have a transparent approach for weighing various considerations (e.g. cost-effectiveness, equity)? Does the agency recommend, or operate, conditional reimbursement/coverage-with-evidence-development schemes? Does the organization distinguish between identifiable subgroups of the patient population when making decisions?

44 Conclusions Benchmarking of HTA entities is currently controversial, but it is feasible New approaches need to be developed that yield unambiguous assessments Benchmarking could either be undertaken by an independent organization, or by the HTA agencies themselves The main focus should be on selfimprovement by the agencies themselves, rather than on external criticism

45