External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis

for the IPPIC Collaborative Network

doi:10.1186/s12916-020-01766-9

External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis

for the IPPIC Collaborative Network

Producción científica: Contribución a una revista › Artículo › revisión exhaustiva

19 Citas (Scopus)

Resumen

Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting.

Idioma original	Inglés
Número de artículo	302
Publicación	BMC Medicine
Volumen	18
N.º	1
DOI	https://doi.org/10.1186/s12916-020-01766-9
Estado	Publicada - 1 dic. 2020

Nota bibliográfica

Publisher Copyright:
© 2020, The Author(s).

Acceder al documento

10.1186/s12916-020-01766-9

Otros archivos y enlaces

Enlace a la publicación en Scopus

Citar esto

@article{95c74fcb39fb4372be1bdc22669fed5a,

title = "External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis",

abstract = "Background: Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting. Methods: IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis. Results: Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model{\textquoteright}s calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%. Conclusions: The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice. Trial registration: PROSPERO ID: CRD42015029349.",

keywords = "External validation, Individual participant data, Pre-eclampsia, Prediction model",

author = "{for the IPPIC Collaborative Network} and Snell, {Kym I.E.} and John Allotey and Melanie Smuk and Richard Hooper and Claire Chan and Asif Ahmed and Chappell, {Lucy C.} and {Von Dadelszen}, Peter and Marcus Green and Louise Kenny and Asma Khalil and Khan, {Khalid S.} and Mol, {Ben W.} and Jenny Myers and Lucilla Poston and Basky Thilaganathan and Staff, {Anne C.} and Smith, {Gordon C.S.} and Wessel Ganzevoort and Hannele Laivuori and Odibo, {Anthony O.} and {Arenas Ram{\'i}rez}, Javier and John Kingdom and George Daskalakis and Diane Farrar and Baschat, {Ahmet A.} and Seed, {Paul T.} and Federico Prefumo and {da Silva Costa}, Fabricio and Henk Groen and Francois Audibert and Jacques Masse and Skr{\aa}stad, {Ragnhild B.} and Kjell Salvesen and Camilla Haavaldsen and Chie Nagata and Rumbold, {Alice R.} and Seppo Heinonen and Askie, {Lisa M.} and Smits, {Luc J.M.} and Vinter, {Christina A.} and Per Magnus and Kajantie Eero and Villa, {Pia M.} and Jenum, {Anne K.} and Andersen, {Louise B.} and Norman, {Jane E.} and Akihide Ohkuchi and Anne Eskild and Illanes, {Sebasti{\'a}n E.}",

note = "Funding Information: This project was funded by the National Institute for Health Research Health Technology Assessment Programme (ref no: 14/158/02). Kym Snell is funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The UK Medical Research Council and Wellcome (grant ref.: 102215/2/13/2) and the University of Bristol provide core support for ALSPAC. This publication is the work of the authors, and ST, RR, KS, and JA will serve as guarantors for the contents of this paper. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. Acknowledgements Funding Information: The authors disclose support from NIHR HTA for the submitted work. LCC reports being Chair of the HTA CET Committee from January 2019. AK reports being a member of the NIHR HTA board. BWM reports grants from Merck; personal fees from OvsEva, Merck, and Guerbet; and other from NHMRC, Guerbet, and Merch, outside the submitted work. GS reports grants and personal fees from GlaxoSmithKline Research and Development Limited, grants from Sera Prognostics Inc., non-financial support from Illumina Inc., and personal fees and non-financial support from Roche Diagnostics Ltd., outside the submitted work. JK reports personal fees from Roche Canada, outside the submitted work. JM reports grants from National Health Research and Development Program, Health and Welfare Canada, during the conduct of the study. JEN reports grants from Chief Scientist Office Scotland, other from GlaxoSmithKline and Dilafor, outside the submitted work. AG reports personal fees from Roche Diagnostics, outside the submitted work. IH reports personal fees from Roche Diagnostics and Thermo Fisher, outside the submitted work. RR reports personal fees from the BMJ, Roche, and Universities of Leeds, Edinburgh, and Exeter, outside the submitted work. Publisher Copyright: {\textcopyright} 2020, The Author(s).",

year = "2020",

month = dec,

day = "1",

doi = "10.1186/s12916-020-01766-9",

language = "English",

volume = "18",

journal = "BMC Medicine",

issn = "1741-7015",

publisher = "BioMed Central Ltd.",

number = "1",

}

TY - JOUR

T1 - External validation of prognostic models predicting pre-eclampsia

T2 - individual participant data meta-analysis

AU - for the IPPIC Collaborative Network

AU - Snell, Kym I.E.

AU - Allotey, John

AU - Smuk, Melanie

AU - Hooper, Richard

AU - Chan, Claire

AU - Ahmed, Asif

AU - Chappell, Lucy C.

AU - Von Dadelszen, Peter

AU - Green, Marcus

AU - Kenny, Louise

AU - Khalil, Asma

AU - Khan, Khalid S.

AU - Mol, Ben W.

AU - Myers, Jenny

AU - Poston, Lucilla

AU - Thilaganathan, Basky

AU - Staff, Anne C.

AU - Smith, Gordon C.S.

AU - Ganzevoort, Wessel

AU - Laivuori, Hannele

AU - Odibo, Anthony O.

AU - Arenas Ramírez, Javier

AU - Kingdom, John

AU - Daskalakis, George

AU - Farrar, Diane

AU - Baschat, Ahmet A.

AU - Seed, Paul T.

AU - Prefumo, Federico

AU - da Silva Costa, Fabricio

AU - Groen, Henk

AU - Audibert, Francois

AU - Masse, Jacques

AU - Skråstad, Ragnhild B.

AU - Salvesen, Kjell

AU - Haavaldsen, Camilla

AU - Nagata, Chie

AU - Rumbold, Alice R.

AU - Heinonen, Seppo

AU - Askie, Lisa M.

AU - Smits, Luc J.M.

AU - Vinter, Christina A.

AU - Magnus, Per

AU - Eero, Kajantie

AU - Villa, Pia M.

AU - Jenum, Anne K.

AU - Andersen, Louise B.

AU - Norman, Jane E.

AU - Ohkuchi, Akihide

AU - Eskild, Anne

AU - Illanes, Sebastián E.

N1 - Funding Information: This project was funded by the National Institute for Health Research Health Technology Assessment Programme (ref no: 14/158/02). Kym Snell is funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The UK Medical Research Council and Wellcome (grant ref.: 102215/2/13/2) and the University of Bristol provide core support for ALSPAC. This publication is the work of the authors, and ST, RR, KS, and JA will serve as guarantors for the contents of this paper. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. Acknowledgements Funding Information: The authors disclose support from NIHR HTA for the submitted work. LCC reports being Chair of the HTA CET Committee from January 2019. AK reports being a member of the NIHR HTA board. BWM reports grants from Merck; personal fees from OvsEva, Merck, and Guerbet; and other from NHMRC, Guerbet, and Merch, outside the submitted work. GS reports grants and personal fees from GlaxoSmithKline Research and Development Limited, grants from Sera Prognostics Inc., non-financial support from Illumina Inc., and personal fees and non-financial support from Roche Diagnostics Ltd., outside the submitted work. JK reports personal fees from Roche Canada, outside the submitted work. JM reports grants from National Health Research and Development Program, Health and Welfare Canada, during the conduct of the study. JEN reports grants from Chief Scientist Office Scotland, other from GlaxoSmithKline and Dilafor, outside the submitted work. AG reports personal fees from Roche Diagnostics, outside the submitted work. IH reports personal fees from Roche Diagnostics and Thermo Fisher, outside the submitted work. RR reports personal fees from the BMJ, Roche, and Universities of Leeds, Edinburgh, and Exeter, outside the submitted work. Publisher Copyright: © 2020, The Author(s).

PY - 2020/12/1

Y1 - 2020/12/1

N2 - Background: Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting. Methods: IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis. Results: Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model’s calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%. Conclusions: The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice. Trial registration: PROSPERO ID: CRD42015029349.

AB - Background: Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting. Methods: IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis. Results: Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model’s calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%. Conclusions: The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice. Trial registration: PROSPERO ID: CRD42015029349.

KW - External validation

KW - Individual participant data

KW - Pre-eclampsia

KW - Prediction model

UR - http://www.scopus.com/inward/record.url?scp=85095396152&partnerID=8YFLogxK

U2 - 10.1186/s12916-020-01766-9

DO - 10.1186/s12916-020-01766-9

M3 - Article

C2 - 33131506

AN - SCOPUS:85095396152

SN - 1741-7015

VL - 18

JO - BMC Medicine

JF - BMC Medicine

IS - 1

M1 - 302

ER -

External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis

Resumen

Nota bibliográfica

Acceder al documento

Otros archivos y enlaces

Huella

Citar esto