Note: The risk of bias by domain corresponds to the highest risk of bias among outcomes by domain.
The overall risk of bias corresponds to the overall highest risk of bias assessed among outcomes.
| Bias | Author's judgement | Support for judgement |
| Confounding |
Serious |
The analysis controlled "age, sex, date, location, comorbidity index plus 17 COVID-19 risk factors” in propensity score matching, but did not control for socioeconomic factors, ethnicity, health-seeking behaviour (although they included ‘healthcare utilization intensity’) or symptoms at the time. |
| Selection of participants into the study |
Low |
The study is based on claims data from the USA. Given the wide population coverage, we do not consider this to introduce important selection bias. |
| Clasification of interventions |
Moderate |
The authors acknowledge that there were likely to be vaccinations not included in the claims data set used as a basis for the study. They performed sensitivity analyses assuming 40% under-recording of vaccinations, which indicated that the unadjusted estimate that we extracted may be roughly a 10% underestimate of vaccine effectiveness. |
| Deviations from intervention |
Low |
No concerns – the study is observational. |
| Missing outcome data |
Moderate |
According to the authors' 'CONSORT' diagram, only those with ‘non-missing’ demographic data (ZIP code, age, sex) were included (3157779/4122506 = 76.6%). Complete case analysis was used and we do not know what was done to address those excluded because they were missing demographic data, which could be related to the outcome (or to intervention status). |
| Measurement of the outcome |
Low |
No particular concerns in this domain - determination of the outcome is unlikely to be biased. |
| Selection of the reported results |
Moderate |
There is no evidence of a protocol/analysis plan, and we have concerns that results could have been selected for reporting because of the findings. |
| Overall risk of bias |
Serious |
|
| Overall comment | In addition to the preprint, the supplementary material files were used in data extraction and risk of bias assessment. Results were extracted for the matched "high Delta incidence states" subgroup (assessed during June-July 2021) since the matched national cohort included periods of low Delta variant. Results were extracted that included a correction for vaccination under-reporting in claims databases; additionally, the authors reported that overall if, "assume 0% under-recording and completely disregard the necessary correction, the VE estimates would decline by about 10 percentage points". Authors used matching to account for confounding, with both risk-set sampling and propensity scores. Prevalence of Delta variant was not reported; stated that it became the dominant strain in overall USA by July 2021.
Concerns about uncontrolled confounding (e.g. by socioeconomic factors, ethnicity) and there is a strong indication that many vaccinations were not represented in the data set. |