Skip to main content
SpringerLink
Log in

Validation of the Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM)

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Background

The Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM) was developed to address study questions pertaining to the cost-effectiveness of treatment alternatives in the care of patients with type 2 diabetes mellitus (T2DM). Naturally, the usefulness of a model is determined by the accuracy of its predictions. A previous version of ECHO-T2DM was validated against actual trial outcomes and the model predictions were generally accurate. However, there have been recent upgrades to the model, which modify model predictions and necessitate an update of the validation exercises.

Objectives

The objectives of this study were to extend the methods available for evaluating model validity, to conduct a formal model validation of ECHO-T2DM (version 2.3.0) in accordance with the principles espoused by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM), and secondarily to evaluate the relative accuracy of four sets of macrovascular risk equations included in ECHO-T2DM.

Methods

We followed the ISPOR/SMDM guidelines on model validation, evaluating face validity, verification, cross-validation, and external validation. Model verification involved 297 ‘stress tests’, in which specific model inputs were modified systematically to ascertain correct model implementation. Cross-validation consisted of a comparison between ECHO-T2DM predictions and those of the seminal National Institutes of Health model. In external validation, study characteristics were entered into ECHO-T2DM to replicate the clinical results of 12 studies (including 17 patient populations), and model predictions were compared to observed values using established statistical techniques as well as measures of average prediction error, separately for the four sets of macrovascular risk equations supported in ECHO-T2DM. Sub-group analyses were conducted for dependent vs. independent outcomes and for microvascular vs. macrovascular vs. mortality endpoints.

Results

All stress tests were passed. ECHO-T2DM replicated the National Institutes of Health cost-effectiveness application with numerically similar results. In external validation of ECHO-T2DM, model predictions agreed well with observed clinical outcomes. For all sets of macrovascular risk equations, the results were close to the intercept and slope coefficients corresponding to a perfect match, resulting in high R 2 and failure to reject concordance using an F test. The results were similar for sub-groups of dependent and independent validation, with some degree of under-prediction of macrovascular events.

Conclusion

ECHO-T2DM continues to match health outcomes in clinical trials in T2DM, with prediction accuracy similar to other leading models of T2DM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. American Diabetes Association. Standards of medical care in diabetes: 2016. Diabetes Care. 2016;39(Suppl. 1):S1–112.

    Google Scholar 

  2. Stratton IM, Adler AI, Neil HA, et al. Association of glycaemia with macrovascular and microvascular complications of type 2 diabetes (UKPDS 35): prospective observational study. BMJ. 2000;321(7258):405–12.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Fowler MJ. Microvascular and macrovascular complications of diabetes. Clin Diabetes. 2008;26(2):77–82.

    Article  Google Scholar 

  4. Inzucchi SE, Bergenstal RM, Buse JB, et al. Management of hyperglycemia in type 2 diabetes, 2015: a patient-centered approach: update to a position statement of the American Diabetes Association and the European Association for the Study of Diabetes. Diabetes Care. 2015;38(1):140–9.

    Article  PubMed  Google Scholar 

  5. UKPDS Study Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33): UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998;352(9131):837–53.

    Article  Google Scholar 

  6. Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices-Modeling Studies. Value Health. 2003;6(1):9–17.

  7. Caro JJ, Briggs AH, Siebert U, Kuntz KM. Modeling good research practices: overview. A report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value Health. 2012;15(6):796–803.

    Article  PubMed  Google Scholar 

  8. American Diabetes Association Consensus Panel. Guidelines for computer modeling of diabetes and its complications. Diabetes Care. 2004;27(9):2262–5.

    Article  Google Scholar 

  9. Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Med Decis Making. 2012;32(5):733–43.

  10. Brown JB, Palmer AJ, Bisgaard P, et al. The Mt. Hood challenge: cross-testing two diabetes simulation models. Diabetes Res Clin Pract. 2000;50(Suppl. 3):S57–64.

    Article  PubMed  Google Scholar 

  11. The Mount Hood. 4 Modeling Group. Computer modeling of diabetes and its complications: a report on the Fourth Mount Hood challenge meeting. Diabetes Care. 2007;30(6):1638–46.

    Article  Google Scholar 

  12. Palmer AJ, Clarke P, Gray A, et al. Computer modeling of diabetes and its complications: a report on the Fifth Mount Hood challenge meeting. Value Health. 2013;16(4):670–85.

    Article  PubMed  Google Scholar 

  13. Eddy DM, Schlessinger L. Validation of the Archimedes diabetes model. Diabetes Care. 2003;26(11):3102–10.

    Article  PubMed  Google Scholar 

  14. Palmer AJ, Roze S, Valentine WJ, et al. Validation of the CORE diabetes model against epidemiological and clinical studies. Curr Med Res Opin. 2004;20(Suppl. 1):S27–40.

    Article  PubMed  Google Scholar 

  15. McEwan P, Foos V, Palmer JL, et al. Validation of the IMS CORE diabetes model. Value Health. 2014;17(6):714–24.

    Article  PubMed  Google Scholar 

  16. Mueller E, Maxion-Bergemann S, Gultyaev D, et al. Development and validation of the Economic Assessment of Glycemic Control and Long-Term Effects of diabetes (EAGLE) model. Diabetes Technol Ther. 2006;8(2):219–36.

    Article  PubMed  Google Scholar 

  17. Willis M, Asseburg C, He J. Validation of economic and health outcomes simulation model of type 2 diabetes mellitus (ECHO-T2DM). J Med Econom. 2013;16(8):1007–21.

    Article  Google Scholar 

  18. McEwan P, Ward T, Bennett H, Bergenheim K. Validation of the UKPDS 82 risk equations within the Cardiff Diabetes Model. Cost Eff Resour Alloc. 2015;13:12.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Hoerger T, Segel J, Zhang P. Validation of the CDC-RTI diabetes cost-effectiveness model: RTI Press publication no. MR-0013-0909. Research Triangle Park (NC): RTI International; 2009. http://www.rti.org/rtipress. Accessed 1 Mar 2013.

  20. Brown JB, Russell A, Chan W, et al. The global diabetes model: user friendly version 3.0. Diabetes Res Clin Pract. 2000;50(Suppl. 3):S15–46.

    Article  PubMed  Google Scholar 

  21. Bagust A, Hopkinson PK, Maier W, Currie CJ. An economic model of the long-term health care burden of type II diabetes. Diabetologia. 2001;44(12):2140–55.

    Article  CAS  PubMed  Google Scholar 

  22. Eastman RC, Javitt JC, Herman WH, et al. Model of complications of NIDDM. I. Model construction and assumptions. Diabetes Care. 1997;20(5):725–34.

    Article  CAS  PubMed  Google Scholar 

  23. Hoerger TJ, Wittenborn JS, Segel JE, et al. A health policy model of CKD. 1. Model construction, assumptions, and validation of health consequences. Am J Kidney Dis. 2010;55(3):452–62.

    Article  PubMed  Google Scholar 

  24. Clarke PM, Gray AM, Briggs A, et al. A model to estimate the lifetime health outcomes of patients with type 2 diabetes: the United Kingdom Prospective Diabetes Study (UKPDS) Outcomes Model (UKPDS no. 68). Diabetologia. 2004;47(10):1747–59.

    Article  CAS  PubMed  Google Scholar 

  25. Hayes AJ, Leal J, Gray AM, et al. UKPDS outcomes model 2: a new version of a model to simulate lifetime health outcomes of patients with type 2 diabetes mellitus using data from the 30 year United Kingdom Prospective Diabetes Study: UKPDS 82. Diabetologia. 2013;56(9):1925–33.

    Article  CAS  PubMed  Google Scholar 

  26. Kengne AP, Patel A, Marre M, et al. Contemporary model for cardiovascular risk prediction in people with type 2 diabetes. Eur J Cardiovasc Prev Rehabil. 2011;18(3):393–8.

    Article  PubMed  Google Scholar 

  27. Zethelius B, Eliasson B, Eeg-Olofsson K, et al. A new model for 5-year risk of cardiovascular disease in type 2 diabetes, from the Swedish National Diabetes Register (NDR). Diabetes Res Clin Pract. 2011;93(2):276–84.

    Article  PubMed  Google Scholar 

  28. Hoerger TJ, Wittenborn JS, Segel JE, et al. A health policy model of CKD. Am J Kidney Dis. 2010;55(3):463–73.

    Article  PubMed  Google Scholar 

  29. Hoerger TJ, Wittenborn JS, Segel JE, et al. A health policy model of CKD: 2. The cost-effectiveness of microalbuminuria screening. Am J Kidney Dis. 2010;55(3):463–73.

    Article  PubMed  Google Scholar 

  30. Eastman RC, Javitt JC, Herman WH, et al. Model of complications of NIDDM. II. Analysis of the health benefits and cost-effectiveness of treating NIDDM with the goal of normoglycemia. Diabetes Care. 1997;20(5):735–44.

    Article  CAS  PubMed  Google Scholar 

  31. Colhoun HM, Betteridge DJ, Durrington PN, et al. Primary prevention of cardiovascular disease with atorvastatin in type 2 diabetes in the Collaborative Atorvastatin Diabetes Study (CARDS): multicentre randomised placebo-controlled trial. Lancet. 2004;364(9435):685–96.

  32. HOPE Study Investigators. Effects of ramipril on cardiovascular and microvascular outcomes in people with diabetes mellitus: results of the HOPE study and MICRO-HOPE substudy. Lancet. 2000;355(9200):253–9.

    Article  Google Scholar 

  33. Klein R, Klein BE, Moss SE, et al. The Wisconsin epidemiologic study of diabetic retinopathy. III. Prevalence and risk of diabetic retinopathy when age at diagnosis is 30 or more years. Arch Ophthalmol. 1984;102(4):527–32.

    Article  CAS  PubMed  Google Scholar 

  34. Klein R, Klein BE, Moss SE, et al. Glycosylated hemoglobin predicts the incidence and progression of diabetic retinopathy. JAMA. 1988;260(19):2864–71.

    Article  CAS  PubMed  Google Scholar 

  35. Zhou H, Isaman DJ, Messinger S, et al. A computer simulation model of diabetes progression, quality of life, and cost. Diabetes Care. 2005;28(12):2856–63.

    Article  PubMed  Google Scholar 

  36. Sasaki A, Uehara M, Horiuchi N, et al. A 15 year follow-up study of patients with non-insulin dependent diabetes mellitus (NIDDM) in Osaka, Japan: long-term prognosis and causes of death. Diabetes Res Clin Pract. 1996;34(1):47–55.

    Article  CAS  PubMed  Google Scholar 

  37. Knopp RH, d’Emden M, Smilde JG, Pocock SJ. Efficacy and safety of atorvastatin in the prevention of cardiovascular end points in subjects with type 2 diabetes: the Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in non-insulin-dependent diabetes mellitus (ASPEN). Diabetes Care. 2006;29(7):1478–85.

    Article  CAS  PubMed  Google Scholar 

  38. Gerstein HC, Miller ME, Byington RP, et al. Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med. 2008;358(24):2545–59.

    Article  CAS  PubMed  Google Scholar 

  39. Cushman WC, Evans GW, Byington RP, et al. Effects of intensive blood-pressure control in type 2 diabetes mellitus. N Engl J Med. 2010;362(17):1575–85.

    Article  PubMed  Google Scholar 

  40. Patel A, MacMahon S, Chalmers J, et al. Intensive blood glucose control and vascular outcomes in patients with type 2 diabetes. N Engl J Med. 2008;358(24):2560–72.

    Article  CAS  PubMed  Google Scholar 

  41. Woodward M, Patel A, Zoungas S, et al. Does glycemic control offer similar benefits among patients with diabetes in different regions of the world? Results from the ADVANCE trial. Diabetes Care. 2011;34(12):2491–5.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Holman RR, Paul SK, Bethel MA, et al. 10-year follow-up of intensive glucose control in type 2 diabetes. N Engl J Med. 2008;359(15):1577–89.

    Article  CAS  PubMed  Google Scholar 

  43. Eeg-Olofsson K, Cederholm J, Nilsson PM, et al. New aspects of HbA1c as a risk factor for cardiovascular diseases in type 2 diabetes: an observational study from the Swedish National Diabetes Register (NDR). J Intern Med. 2010;268(5):471–82.

    Article  CAS  PubMed  Google Scholar 

  44. Kelly PJ, Clarke PM, Hayes AJ, et al. Predicting mortality in people with type 2 diabetes mellitus after major complications: a study using Swedish National Diabetes Register data. Diabetic Med. 2014;31(8):954–62.

    Article  CAS  PubMed  Google Scholar 

  45. Wing RR, Bolin P, Brancati FL, et al. Cardiovascular effects of intensive lifestyle intervention in type 2 diabetes. N Engl J Med. 2013;369(2):145–54.

    Article  CAS  PubMed  Google Scholar 

  46. National Institute for Health and Care Excellence. Dapagliflozin in combination therapy for treating type 2 diabetes: technology appraisal guidance (TA288). National Institute for Health and Care Excellence; 2013. https://www.nice.org.uk/guidance/ta288/resources/dapagliflozin-in-combination-therapy-for-treating-type2-diabetes-82600679642821.

  47. Shepherd J, Blauw GJ, Murphy MB, et al. Pravastatin in elderly individuals at risk of vascular disease (PROSPER): a randomised controlled trial. Lancet. 2002;360(9346):1623–30.

    Article  CAS  PubMed  Google Scholar 

  48. Dent JB, Blackie MJ. Systems simulation in agriculture. Netherlands: Springer, Netherlands; 1979.

    Book  Google Scholar 

  49. Thornton PK, Hansen JW. A note on regressing real-world data on model output. Agric Syst. 1996;50(4):411–4.

    Article  Google Scholar 

  50. Tofallis C. A better measure of relative prediction accuracy for model selection and model estimation. J Oper Res Soc. 2015;66(8):1352–62.

    Article  Google Scholar 

  51. Bellocchi G, Rivington M, Donatelli M, Matthews K. Validation of biophysical models: issues and methodologies: a review. Agron Sustain Dev. 2010;30(1):109–30.

    Article  Google Scholar 

  52. Mayer DG, Butler DG. Theoretical modelling aspects statistical validation. Ecol Model. 1993;68(1):21–32.

    Article  Google Scholar 

  53. Thompson PA. Evaluation of the M-competition forecasts via log mean squared error ratio. Int J Forecast. 1991;7(3):331–4.

    Article  Google Scholar 

  54. Lundqvist A, Steen Carlsson K, Johansen P, et al. Validation of the IHE cohort model of type 2 diabetes and the impact of choice of macrovascular risk equations. PLoS One. 2014;9(10):e110235.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Neslusan C, Teschemaker A, Johansen P, et al. Cost-effectiveness of canagliflozin versus sitagliptin as add-on to metformin in patients with type 2 diabetes mellitus in Mexico. Value Health Reg Issues. 2015;8:8–19.

    Article  Google Scholar 

  56. Sabapathy S, Neslusan C, Yoong K, et al. Cost-effectiveness of canagliflozin versus sitagliptin when added to metformin and sulfonylurea in type 2 diabetes in Canada. J Pop Ther Clin Pharmacol. 2016;23(2):e151–68.

    Google Scholar 

  57. Vemer P, Corro Ramos I, van Voorn GA, et al. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. PharmacoEconomics. 2016;34(4):349–61.

    Article  CAS  PubMed  Google Scholar 

  58. National Institute for Health and Care Excellence. Canagliflozin in combination therapy for treating type 2 diabetes: technology appraisal guidance (TA315). National Institute for Health and Care Excellence; 2014. https://www.nice.org.uk/guidance/ta288/resources/dapagliflozin-in-combination-therapy-for-treating-type2-diabetes-82600679642821.

  59. Canadian Drug Expert Committee. CDEC final recommendation: canagliflozin (Invokana, Janssen Inc.). 2015. https://www.cadth.ca/media/cdr/complete/cdr_complete_SR0370_Invokana_Jan-19_15.pdf. Accessed 7 Nov 2016.

  60. SMC. Canagliflozin, 100 mg and 300 mg film-coated tablets (Invokana). Scottish Medicines Consortium; 2014 May 9. https://ww.cottishmedicines.rg.k/iles/dvice/__Scottish_Medicine_Consortium_Web_Data_Audit_advice_Advice_by_Year_2014_No.6_-_June_2014_canaglaflozin__Invokana__FINAL_May_2014_for_website.pdf.

  61. NCPE. Cost effectiveness of canagliflozin (Invokana®) for adults with type 2 diabetes mellitus to improve glycaemic control as monotherapy or add-on therapy with other anti-hyperglycaemic agents including insulin, when these, together with diet and exercise, do not provide adequate glycaemic control. 2014. http://www.ncpe.ie/wp-content/uploads/2013/04/Summary-Canagliflozin2.pdf. Accessed 7 Nov 2016.

  62. Mount Hood Challenge. https://sites.google.com/site/mounthoodchallenge/. Accessed 7 Nov 2016.

  63. Mount Hood Challenge. Economics, simulation modelling, and diabetes: 2016 Mount Hood Challenge. http://www.mthooddiabeteschallenge.com/. Accessed 7 Nov 2016.

  64. Charokopou M, Sabater FJ, Townsend R, et al. Methods applied in cost-effectiveness models for treatment strategies in type 2 diabetes mellitus and their use in health technology assessments: a systematic review of the literature from 2008 to 2013. Curr Med Res Opin. 2016;32(2):207–18.

    Article  CAS  PubMed  Google Scholar 

  65. Jaime Caro J, Eddy DM, Kan H, et al. Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):174–82.

    Article  CAS  PubMed  Google Scholar 

  66. Marcus AH, Elias RW. Some useful statistical methods for model validation. Environ Health Perspect. 1998;106(Suppl. 6):1541–50.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Eddy D, Cohen M-D, Shum K, Dziuba J. Validation methodology and results: ARCHeS Simulator 2.5. San Francisco (CA): Archimedes: Quantifying Healthcare; 2013.

Download references

Acknowledgments

Editorial assistance was provided by Shannon O’Sullivan, ELS, of MedErgy, and was funded by Janssen Global Services, LLC. The authors thank Emelie Toresson Grip, MSc, of the Swedish Institute for Health Economics for assistance with the analysis.

Author Contributions

MW and CA were involved in the concept and design of the study. MW, CA, PJ, and AN were involved in the analysis and interpretation of the results. PJ and AN managed the data. MW, CA, and PJ drafted the manuscript. MW acts as the overall guarantor of the manuscript.

Author information

Authors and Affiliations

  1. The Swedish Institute for Health Economics, Box 2127, SE-220 02, Lund, Sweden

    Michael Willis, Pierre Johansen, Andreas Nilsson & Christian Asseburg

Authors
  1. Michael Willis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre Johansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andreas Nilsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christian Asseburg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Willis.

Ethics declarations

Funding

Janssen Global Services, LLC. Provided financial funding.

Conflict of interest

This study was funded by Janssen Global Services, LLC. MW, PJ, CA, and AN are employees of the Swedish Institute for Health Economics, which has provided consulting services for governmental bodies, academic institutions, and commercial life science enterprises. The authors had full independent control over the contents of the manuscript.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 1482 kb)

Supplementary material 2 (PDF 210 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Willis, M., Johansen, P., Nilsson, A. et al. Validation of the Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM). PharmacoEconomics 35, 375–396 (2017). https://doi.org/10.1007/s40273-016-0471-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-016-0471-3

Keywords

Search

Navigation