Did publication urgency during the pandemic trade-off for epidemiological rigorousness? Results from an umbrella review

Background: Scientific studies related to COVID-19 are pivotal for understanding patient characteristics, changing disease patterns and treatments effectiveness and safety. During the pandemic, the urgent need for constantly updated evidence synthesis to inform public policy placed systematic reviews (SRs) at the centre of COVID-19 research. However, the publication urgency from the scientific community appeared to trade off the epidemiological input capturing the data complexity, drivers, and impact of the pandemic.

Objectives: We aimed to evaluate if previous SLRs on predictors of COVID-19 severity and disease progression have incorporated detailed epidemiological considerations in terms of appropriateness of study designs and impact of different types of biases.

Methods: An umbrella review was conducted following PRISMA and Cochrane guidelines to systematically identify and summarise evidence from previously conducted SRs on predictors in COVID-19 outcomes. Several electronic databases were searched from 2020 onwards with English language restrictions. Dual data screening and extraction was guided by a pre-defined PROSPERO registered protocol and each publication was assessed independently by two reviewers. Conflicts were resolved by a senior third reviewer. Results were synthesized qualitatively across the identified SRs.

Results: Across databases, searches retrieved 4564 publications. Abstract and full text screening is ongoing; at 60% level completed, 20 SLRs were identified to investigate the role of single or multiple factors in COVID-19 progression. All SLRs followed clear search strategy with well defined inclusion and exclusion criteria. 90% of included studies in each SLR were quality assessed by established tools/ checklists but methodological considerations specific to COVID-19 disease in terms of study and outcomes definitions and biases linked to changing nature of pandemic were not consistently addressed (less than 50%). Although meta-analysis was performed in most SLRs, only a few of them used quality criteria to inform which studies could provide reliable results for a quantitative synthesis.

Conclusions: Lack of assessment and consideration of potential flaws and biases within the included studies in SRs limits the validity of a review and the generalisability of its findings, thus compromising trust in evidence summaries. Extra caution should be applied when interpreting the results of SLRs published during pandemic.