|Year : 2022 | Volume
| Issue : 4 | Page : 222-227
Quality of systematic reviews/meta-analyses in coronavirus disease 2019 and venous thromboembolism: An analysis using a measurement tool to assess systematic reviews-2
Sultan Alkhamesi1, Abdullah Almohammadi2, Hatem Alahwal2, Ahmed Barefah2, Salem Bahashwan2, Osman Radhwi2
1 Department of Hematology, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
2 Department of Hematology, Faculty of Medicine, King Abdulaziz University; Hematology Research Unit, King Fahd Medical Research Center, King Abdulaziz University, Jeddah, Saudi Arabia
|Date of Submission||18-Apr-2022|
|Date of Decision||26-Apr-2022|
|Date of Acceptance||06-Jun-2022|
|Date of Web Publication||18-Oct-2022|
Dr. Osman Radhwi
Department of Hematology, Faculty of Medicine, King Abdulaziz University, P.O. Box: 80200, Zip Code: 21589, Jeddah
Source of Support: None, Conflict of Interest: None
BACKGROUND: Many systematic reviews (SRs) and meta-analyses (MAs) have been published regarding the incidence of venous thromboembolism (VTE) in coronavirus disease 2019 (COVID-19) patients, which is acknowledged as high. Therefore, we aimed to assess the methodological quality of any SR/MA involving VTE and COVID-19.
METHODS: This is a meta-epidemiological study evaluating the methodological quality of SR/MA. A structured search in PubMed and Web of Science was done for all SRs/MAs in VTE and COVID-19 from inception till April 29, 2021. We evaluated the methodological quality using A Measurement Tool to Assess SRs-2 (AMSTAR-2) checklist.
RESULTS: We included 29 SRs/MAs where only one SR/MA was rated as high (3.44%), 12 (41.37%) were rated as moderate, and the rest were rated as low/critically low (55.17%). No statistical significance was observed in Spearman's correlation analysis between the rate of methodological quality and the number of authors, searching period, publication month, and publishing journal's impact factor.
CONCLUSIONS: The number of published SRs/MAs in the VTE and COVID-19 subject is rising rapidly. The readers should pay attention to the low methodological quality of most published SRs/MAs.
Keywords: A Measurement Tool to Assess Systematic Reviews-2, coronavirus disease 2019, methodological quality, systematic review, venous thromboembolism
|How to cite this article:|
Alkhamesi S, Almohammadi A, Alahwal H, Barefah A, Bahashwan S, Radhwi O. Quality of systematic reviews/meta-analyses in coronavirus disease 2019 and venous thromboembolism: An analysis using a measurement tool to assess systematic reviews-2. J Appl Hematol 2022;13:222-7
|How to cite this URL:|
Alkhamesi S, Almohammadi A, Alahwal H, Barefah A, Bahashwan S, Radhwi O. Quality of systematic reviews/meta-analyses in coronavirus disease 2019 and venous thromboembolism: An analysis using a measurement tool to assess systematic reviews-2. J Appl Hematol [serial online] 2022 [cited 2022 Dec 3];13:222-7. Available from: https://www.jahjournal.org/text.asp?2022/13/4/222/358709
| Background|| |
A novel coronavirus emerged in Wuhan, China, in late December 2019 after being found in multiple patients who presented with unexplained viral pneumonia. The World Health Organization (WHO) officially called this disease coronavirus disease 2019 (COVID-19) on February 11, 2020. A dramatic global spread urged the WHO to declare COVID-19 a pandemic on March 11, 2020. Since the outbreak, COVID-19 has become a significant health and economic burden worldwide. More than 500 million cases have been confirmed worldwide with unprecedented numbers, and the cumulative deaths have exceeded 6,000,000. Saudi Arabia has been affected with the viral spread, and its cumulative cases since the start of the pandemic were approximately 750,000 with 9000 deaths.
Venous thromboembolism (VTE) is a complex, multifactorial disease with several potential factors that could be acquired or hereditary. VTE is a sequential entity including deep vein thrombosis (DVT), thrombus in transit, and acute pulmonary embolism (PE). It is a leading cause of morbidity and mortality worldwide, with 500–800 cases/million/year and sudden death in 10% of the patients.,
Up to a third of the patients with COVID-19 developed thrombotic events, mainly pulmonary emboli, and are usually associated with more severe disease and a predictor of mortality. It is more commonly observed in intensive care patients and cases of critical illness.
Since the start of the COVID-19 pandemic, a concurrent unrivaled pandemic of published articles related to this disease peaked at up to 2400 articles in a single day on June 5, 2020. In December 2020, more than 50,000 full-text articles were documented in the WHO global literature, and the number is expanding by the moment. Many quality assessments of systematic reviews (SRs) or meta-analyses (MAs) published in the COVID-19 era found that despite heavy publishing, most SRs/MAs have poor methodological quality.,,
SRs and MAs are conventionally considered at the top of the pyramid in evidence-based medicine, thus significantly influencing clinicians' decision-making. However, there is a growing concern that the validity of SR and MA variation would affect its results. Therefore, it is necessary to assess the quality of the published SR/MA as the difference in methodological quality level can threaten their validity.,
A newer version of A Measurement Tool to Assess SR-2 (AMSTAR-2) can assess nonrandomized studies of interventions (NRSIs) in addition to its original purpose to assess the quality of SR or MA of randomized clinical trials (RCTs). AMSTAR-2 is not made to generate an overall score. A high score may not necessarily reflect high quality, and critical weaknesses may be disguised in specific domains, such as poor literature search or lack of prior protocol. To make an overall rating of SR, it is essential to consider flaws in critical domains, which may highly affect the quality of a SR. All steps in evaluating SRs and MAs are essential, but seven domains are thought to be more critical than others [Figure 1]. The overall rating generated would be high, moderate, low, or critically low [Figure 1].
|Figure 1: AMSTAR-2 critical domains and an explanatory way of rating the overall confidence in the results of the review. AMSTAR-2: A Measurement Tool to Assess Systematic Reviews-2|
Click here to view
This study aimed to assess the methodological quality of SR/MA for VTE in COVID-19 patients from inception till April 29, 2021, using the AMSTAR-2 tool.
| Methods|| |
Two authors (OR and SK) independently performed a systematic literature search in PubMed and Web of Science from inception until April 29, 2021. Search terms were “COVID-19” or “SARS-CoV-2” or “Novel Coronavirus 2019” and “venous thromboembolism” or “deep vein thrombosis,” “pulmonary embolism,” “thrombosis,” “thromboembolic.” To eliminate irrelevant searches, the titles and abstracts of the found papers were evaluated. The entire texts of the remaining articles were read to see if they met our qualifying requirements. Additional SR/MA was found by looking through the bibliographies of the retrieved publications.
Inclusion and exclusion criteria
The included SR/MA had to discuss the incidence, prevalence, and outcomes of VTE in COVID-19 patients or the use of prophylactic and therapeutic anticoagulants in COVID-19 patients. On the other hand, SR/MA was excluded if it was related to nonhuman studies, had insufficient information (e.g., conference abstracts), or was published in languages other than English.
Study selection and data extraction
According to the inclusion and exclusion criteria, two reviewers (OR and SK) independently selected studies and resolved any conflict by involving the third reviewer (AA) in judgment. First, all related literature was uploaded by the two review authors to a shared Google Drive. Second, according to the inclusion and exclusion criteria, they screened the titles and abstracts of all identified studies. Third, all relevant articles were retrieved in full text and assessed again for the inclusion criteria. Finally, with the aid of the online AMSTAR-2 tool (https://amstar.ca/Amstar_Checklist.php), the same reviewers extracted relevant data to a predesigned data collection sheet. The following items were retrieved from the complete text: journal name, first author's name, number of authors, number of authors with a conflict of interest, country, publishing journal's impact factor, number of searching databases, kind of included study (RCT or NRSI), searching period, and publication quarter, defined as the quarter of the year which the article was published in.
Data synthesis and analysis
Summary statistics were described for all variables. Continuous variables were described as medians with range. Categorical variables were described using a two-way table. The one-way analysis of variance and Fisher's exact test were used for both continuous and categorical variables, respectively. We conducted a sensitivity analysis excluding articles not intended for MA. We used Stata ver. 17.0 (StataCorp LLC, College Station, Texas, United States of America). P < 0.05 was considered statistically significant.
Because this study used only open data, ethics approval was not applicable.
| Results|| |
A total of 4826 records were identified by searching. However, after applying the steps for screening the abstracts and assessing the full texts, only 29 records went into the final step of the methodological assessment.
The essential characteristics of the involved studies are presented in [Table 1]. None of the included SRs/MAs were published in the Cochrane Database of SRs. All SRs/MAs included only NRSI. Approximately half of the studies were published in Europe, 12 studies were in Asia, 5 studies were in North America, and 1 study in Africa. All included SRs/MAs were published in English languages. Concerning study design, none of the SRs/MAs included RCT. Twenty-one studies were MAs (72.41%). Most of the included SRs/MAs discussed the incidence of VTE in hospitalized COVID-19 patients and the use of anticoagulants. In a single SR/MA, three authors or less conducted 3 (10.34%) SRs/MAs, 12 reviews (41.37%) were conducted by 4–6 authors, and 14 reviews had 7 or more authors involved (48.27%). Eight of the SRs/MAs (27.58%) were funded by governmental institutions, while the rest were self-funded, and none of the rest was funded by pharmaceutical companies. Three reviews (10.34%) reported a conflict of interest among their authors.
The interrater agreement on the items listed for the assessment in the AMSTAR-2 was 93%. [Table 2] shows a concise result of the assessment. In addition, a detailed AMSTAR-2 assessment and overall grades for each study are listed in [Supplementary Table 1] and [Supplementary Table 2].
|Table 2: Methodological quality assessment for included systematic reviews/meta-analyses (A Measurement Tool to Assess Systematic Reviews-2) - concise results|
Click here to view
In general, the research questions and inclusion criteria were incorporated into the Population, Intervention, Comparison, and Outcomes for most of the included SRs/MAs (item 1: 96.55%); a comprehensive literature strategy was used (item 4: 96.55%), duplicated the study selection and data extraction (items 5 and 6: 93.10% for both items), and all SRs/MAs described the included studies with the essential data (item 8: 100%).
A total of five SRs/MAs provided a list of excluded studies (item 7: 17.1%), with only one SR/MA justifying exclusion. They were, on the other hand, more likely to offer a plausible explanation for heterogeneity (item 14: 89.65%), consider the risk of bias (ROB) in individual studies when explaining the results of the review (item 13: 58.62%), and mention the funding sources they received when conducting the review (item 16: 100%). However, they were less likely to extract the individual study's funding information (item 10: 6.89%).
All went for sensitivity or subgroup analysis when evaluating SR with MA (item 11: 100%), used suitable statistical procedures to determine the impact of ROB (item 12: 63.63%), and performed a reasonable investigation of small study bias (item 15: 77.27%).
Furthermore, before the start of the review, a few SRs/MAs reported a preexisting protocol and explained any significant change from it (item 2: 31.03%). In the ROB assessment for SR/MA, including both RCT and NRSI, nine studies assessed the necessary domain of ROB (item 9: 37.93%).
In addition, a radar graph was plotted for each critical and noncritical item listed in the AMSTAR-2 checklist [Figure 2] and [Figure 3]. In the overall confidence of the review [Figure 4], one SR/MA was rated as high (3.44%), 12 were rated as moderate (41.37%), 4 were rated as low (13.79%), and 12 were rated as critically low (41.37%).
|Figure 2: Recorded responses of critical items in percentages (yes and partial yes were counted as positive responses)|
Click here to view
|Figure 3: Recorded responses of noncritical items in percentages (yes and partial yes were counted as positive responses)|
Click here to view
|Figure 4: AMSTAR-2 overall grade distribution. AMSTAR: A Measurement Tool to Assess Systematic Reviews|
Click here to view
Characteristics and methodological quality
On Fisher's exact analysis [Supplementary Table 3], there was no statistically significant difference found between the overall quality grade of the review and the publication quarter (P = 0.180), the incorporation of MA (P = 0.542), the number of involved authors (P = 0.154), the number of searched databases (P = 0.343), and the impact factor of the publishing journal (P = 0.430).
| Discussion|| |
Our study is the first epidemiological study in the literature, evaluating the methodological quality of SR/MA dealing with the incidence, prevalence, and prognosis of VTE and the use of anticoagulants in COVID-19 patients. Using the AMSTAR-2 tool, we found that most of the included SRs/MAs were rated low or critically low for their critical flaws.
There was no statistically significant difference between the quality of the SR/MA and other relevant competing variables in our analysis (i.e., publication quarter, a MA study, the number of involved authors, or searched databases). This can be partially related to the small sample size of 29 SRs/MAs. On considering methodological quality, registering a protocol before starting the search phase improves the reliability and the validity of the SR/MA. While most SRs/MAs provided a list of included research, failing to report a list of excluded studies or relying solely on brief generalized remarks for excluded research would compromise the transparency, validity, and purported quality of SR/MA results. We observed a better quality in SR/MA when it explained its approach to the ROB in the discussion.
In addition, none of the SRs/MAs extracted the funding source of individual studies. Our study did not find that SRs/MAs published in journals with a higher impact factor were necessary for higher quality. Due to the nature of the emerging discussed subject, journals were probably more tolerated.
AMSTAR-2 was more acquainted with other methodological quality evaluation methods in the literature, such as ROB in SR (ROBIS). However, we highlight an apparent knowledge gap in which particular ROB calculators for nonrandomized studies are currently evolving.
Our study primarily focuses on the methodological quality of the listed systemic review using AMSTAR-2. Therefore, an in-depth discussion of individual studies was not provided.
Strengths and limitations
Despite the advantages of this study in evaluating the methodological and reporting quality, which is of practical value for researchers, physicians, and policymakers, our study also has limitations:
- Capturing all evidence in the COVID-19 era is difficult as the field rapidly increases. Our search date is up to April 29, 2021, and more SRs are being published in the meantime
- None of the captured SRs/MAs had RCTs, and the AMSTAR-2 tool was developed to analyze the SRs of interventional nature
- Furthermore, we did not assess and critically appraise the included SR/MA using other important tools such as ROB in SRs or diagnostic accuracy such as AMSTARII-DTA extension.
| Conclusion|| |
Most SRs/MAs about VTE and anticoagulation in COVID-19 patients had poor methodological quality, especially in protocol registration and providing a list of excluded studies, hence the critical importance of considering the methodological quality when interpreting the results of SR/MA. Therefore, we encourage journals to imply strict SR/MA methodological quality policies, including implantation of registered protocol, ROB assessment, and following PRISMA guidelines.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Ministry of Health-Kingdom of Saudi Arabia. Saudi COVID-19 Dashboard. Available from: https://covid19.moh.gov.sa/
. [Last accessed on 2022 Apr 13].
Heit JA, Spencer FA, White RH. The epidemiology of venous thromboembolism. J Thromb Thrombolysis 2016;41:3-14.
Alias S, Lang IM. Coagulation and the vessel wall in pulmonary embolism. Pulm Circ 2013;3:728-38.
Middeldorp S, Coppens M, van Haaps TF, Foppen M, Vlaar AP, Müller MC, et al.
Incidence of venous thromboembolism in hospitalized patients with COVID-19. J Thromb Haemost 2020;18:1995-2002.
Ozsu S, Gunay E, Konstantinides SV. A review of venous thromboembolism in COVID-19: A clinical perspective. Clin Respir J 2021;15:506-12.
Wurth R, Hajdenberg M, Barrera FJ, Shekhar S, Copacino CE, Moreno-Peña PJ, et al.
Scoping review of COVID-19-related systematic reviews and meta-analyses: Can we really have confidence in their results? Postgrad Med J 2022;98:372-9.
Li Y, Cao L, Zhang Z, Hou L, Qin Y, Hui X, et al
. Reporting and methodological quality of COVID-19 systematic reviews needs to be improved: An evidence mapping. J Clin Epidemiol 2021;135:17-28.
Pericàs JM, Torrallardona-Murphy O, Arenas A, Valero H, Nicolás D. Profile and quality of published reviews on COVID-19. Eur J Clin Invest 2020;50:e13293.
McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-based medicine and the practicing clinician. J Gen Intern Med 1999;14:236-42.
Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med 2016;21:125-7.
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al
. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008.
Henrina J, Santosa Putra IC, Cahyadi I, Lawrensia S, Hadi Gunawan HF, Cahyadi A, et al
. Clinical characteristics and outcomes of venous thromboembolism in patients hospitalized for COVID-19: Systematic review and meta-analysis. Thromb Updat 2021;2:100037.
Boonyawat K, Chantrathammachart P, Numthavaj P, Nanthatanti N, Phusanti S, Phuphuakrat A, et al
. Incidence of thromboembolism in patients with COVID-19: A systematic review and meta-analysis. Thromb J 2020;18:34.
Zhang C, Shen L, Le KJ, Pan MM, Kong LC, Gu ZC, et al.
Incidence of venous thromboembolism in hospitalized coronavirus disease 2019 patients: A systematic review and meta-analysis. Front Cardiovasc Med 2020;7:151.
Jenner WJ, Kanji R, Mirsadraee S, Gue YX, Price S, Prasad S, et al
. Thrombotic complications in 2928 patients with COVID-19 treated in intensive care: A systematic review. J Thromb Thrombolysis 2021;51:595-607.
Jiménez D, García-Sanchez A, Rali P, Muriel A, Bikdeli B, Ruiz-Artacho P, et al
. Incidence of VTE and bleeding among hospitalized patients with coronavirus disease 2019: A systematic review and meta-analysis. Chest 2021;159:1182-96.
Maldonado E, Tao D, Mackey K. Antithrombotic therapies in COVID-19 disease: A systematic review. J Gen Intern Med 2020;35:2698-706.
Mohamed MF, Al-Shokri SD, Shunnar KM, Mohamed SF, Najim MS, Ibrahim SI, et al.
Prevalence of venous thromboembolism in critically Ill COVID-19 patients: Systematic review and meta-analysis. Front Cardiovasc Med 2020;7:598846.
Nopp S, Moik F, Jilma B, Pabinger I, Ay C. Risk of venous thromboembolism in patients with COVID-19: A systematic review and meta-analysis. Res Pract Thromb Haemost 2020;4: 1178-91.
Patell R, Chiasakul T, Bauer E, Zwicker JI. Pharmacologic thromboprophylaxis and thrombosis in hospitalized patients with COVID-19: A pooled analysis. Thromb Haemost 2021;121:76-85.
Zhang R, Ni L, Di X, Wang X, Ma B, Niu S, et al
. Systematic review and meta-analysis of the prevalence of venous thromboembolic events in novel coronavirus disease-2019 patients. J Vasc Surg Venous Lymphat Disord 2021;9:289-98.e5.
Suh YJ, Hong H, Ohana M, Bompard F, Revel MP, Valle C, et al
. Pulmonary embolism and deep vein thrombosis in COVID-19: A systematic review and meta-analysis. Radiology 2021;298:E70-80.
Wijaya I, Andhika R, Huang I. The use of therapeutic-dose anticoagulation and its effect on mortality in patients with COVID-19: A systematic review. Clin Appl Thromb Hemost 2020;26:144-56.
Wu T, Zuo Z, Yang D, Luo X, Jiang L, Xia Z, et al
. Venous thromboembolic events in patients with COVID-19: A systematic review and meta-analysis. Age Ageing 2021;50:284-93.
Chi G, Lee JJ, Jamil A, Gunnam V, Najafi H, Memar Montazerin S, et al
. Venous thromboembolism among hospitalized patients with COVID-19 undergoing thromboprophylaxis: A systematic review and meta-analysis. J Clin Med 2020;9:2489.
Kefale B, Tegegne GT, Degu A, Tadege M, Tesfa D. Prevalence and risk factors of thromboembolism among patients with coronavirus disease-19: A systematic review and meta-analysis. Clin Appl Thromb Hemost 2020;26:1-3.
Kunutsor SK, Beswick AD, Whitehouse MR, Blom AW. Systematic review of risk prediction scores for venous thromboembolism following joint replacement. Thromb Res 2018;168:148-55.
Porfidia A, Valeriani E, Pola R, Porreca E, Rutjes AW, Di Nisio M. Venous thromboembolism in patients with COVID-19: Systematic review and meta-analysis. Thromb Res 2020;196:67-74.
Birkeland K, Zimmer R, Kimchi A, Kedan I. Venous thromboembolism in hospitalized COVID-19 patients: Systematic review. Interact J Med Res 2020;9:e22768.
Birocchi S, Manzoni M, Podda GM, Casazza G, Cattaneo M. High rates of pulmonary artery occlusions in COVID-19. A meta-analysis. Eur J Clin Invest 2021;51:e13433.
Wang C, Zhang H, Zhou M, Cheng Y, Ye L, Chen J, et al
. Prognosis of COVID-19 in patients with vein thrombosis: A systematic review and meta-analysis. Eur Rev Med Pharmacol Sci 2020;24:10279-85.
Di Minno A, Ambrosino P, Calcaterra I, Di Minno MN. COVID-19 and venous thromboembolism: A meta-analysis of literature studies. Semin Thromb Hemost 2020;46:763-71.
Fontana P, Casini A, Robert-Ebadi H, Glauser F, Righini M, Blondon M. Venous thromboembolism in COVID-19: Systematic review of reported risks and current guidelines. Swiss Med Wkly 2020;150:w20301.
Hasan SS, Radford S, Kow CS, Zaidi ST. Venous thromboembolism in critically ill COVID-19 patients receiving prophylactic or therapeutic anticoagulation: A systematic review and meta-analysis. J Thromb Thrombolysis 2020;50:814-21.
Liu Y, Cai J, Wang C, Jin J, Qu L. A systematic review and meta-analysis of incidence, prognosis, and laboratory indicators of venous thromboembolism in hospitalized patients with coronavirus disease 2019. J Vasc Surg Venous Lymphat Disord 2021;9:1099-111.e6.
Lu YF, Pan LY, Zhang WW, Cheng F, Hu SS, Zhang X, et al
. A meta-analysis of the incidence of venous thromboembolic events and impact of anticoagulation on mortality in patients with COVID-19. Int J Infect Dis 2020;100:34-41.
Malas MB, Naazie IN, Elsayed N, Mathlouthi A, Marmor R, Clary B. Thromboembolism risk of COVID-19 is high and associated with a higher risk of mortality: A systematic review and meta-analysis. EClinicalMedicine 2020;29:100639.
Roncon L, Zuin M, Barco S, Valerio L, Zuliani G, Zonzin P, et al
. Incidence of acute pulmonary embolism in COVID-19 patients: Systematic review and meta-analysis. Eur J Intern Med 2020;82:29-37.
Sarfraz A, Sarfraz Z, Razzack AA, Patel G, Sarfraz M. Venous thromboembolism, corticosteroids and COVID-19: A systematic review and meta-analysis. Clin Appl Thromb Hemost 2021;27:1-7.
Servante J, Swallow G, Thornton JG, Myers B, Munireddy S, Malinowski AK, et al
. Haemostatic and thrombo-embolic complications in pregnant women with COVID-19: A systematic review and critical analysis. BMC Pregnancy Childbirth 2021;21:108.
Perry R, Whitmarsh A, Leach V, Davies P. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev 2021;10:273.
[Figure 1], [Figure 2], [Figure 3], [Figure 4]
[Table 1], [Table 2]