Note: The risk of bias by domain corresponds to the highest risk of bias among outcomes by domain.
The overall risk of bias corresponds to the overall highest risk of bias assessed among outcomes.
| Bias | Author's judgement | Support for judgement |
| Confounding |
Serious |
The analysis matched for some of the pre-defined confounders (age, sex, ethnicity/race, comorbidities, calendar time). Geographic location and health-seeking behaviour were similar due to restriction of the study design. Uncontrolled confounding remains likely, for example by socioeconomic status and symptoms at time of vaccination. |
| Selection of participants into the study |
Moderate |
The study used a test-negative design. This has the potential to suffer from selection bias by being restricted to individuals getting a test, although the issues are not yet well understood. Individuals who received doses from more than one manufacturer or only received one vaccine dose were excluded from the analysis. |
| Clasification of interventions |
Moderate |
Some concern in this domain –vaccination may have not been recorded for all participants. |
| Deviations from intervention |
Low |
No concerns - the study is observational. |
| Missing outcome data |
Moderate |
Some concern in this domain due to missing data on ethnicity resulting in 28% of vaccinated excluded from the matched analysis. |
| Measurement of the outcome |
Moderate |
Controls would be identified as infected only if they seek testing (Moderate risk of bias for infection outcome). No particular concerns (Low risk of bias) for severe disease and death. |
| Selection of the reported results |
Moderate |
There is no evidence of an analysis plan, and we have concerns that results could have been selected for reporting because of the findings. |
| Overall risk of bias |
Serious |
|
| Overall comment | In addition to the preprint, the supplementary material were used in data extraction and risk of bias assessment. Authors used matching and adjustment to account for confounding. 78.8% received Pfizer/BNT162b2, while 21.2% received Moderna/mRNA-1273. The infections table (eTable 2) appears to have a date typographical error; it states after "June" 18, 2021 but the text and other figures indicate this should probably be after "January" 18, 2021.
Concerns over uncontrolled confounding, missing data on a confounder and (for infection outcomes) the possibility that controls would only be identified if they seek testing; in addition there are uncertainties over the possibility of selection bias in a test-negative design. |