EditversePRISMA 2020 flow diagrams: the counting errors reviewers catch
A PRISMA 2020 flow diagram is a set of subtractions that must balance. A worked reconciliation through every box, with databases and registers kept apart from other sources, and the ten errors to check for.
Key points
- The PRISMA 2020 flow diagram counts three different things: records, reports and studies. Most errors come from switching between them without saying so.
- The template has two columns: studies identified through databases and registers, and studies identified through other methods such as websites, organisations and citation searching. They are counted separately and meet only at the final box.
- Each box equals the box above it minus the box to its side. Five subtractions in the left column and two in the right must all balance.
- Reasons for excluding reports must sum to the total excluded, which means assigning one primary reason per report.
- The final box gives both the number of studies and the number of reports. They differ whenever one study was published in more than one report.
What PRISMA 2020 asks for
Item 16a of the PRISMA 2020 checklist asks authors to describe the results of the search and selection process, from the number of records identified to the number of studies included, ideally with a flow diagram.1 The explanation and elaboration paper lists what has to be counted: records identified; records excluded before screening, for example as duplicates or because an automation tool marked them ineligible; records screened; records excluded after screening of titles or titles and abstracts; reports retrieved for detailed evaluation; potentially eligible reports that could not be retrieved; retrieved reports that did not meet the inclusion criteria, with the main reasons; and the number of studies and reports included.2
The PRISMA website provides four templates: for new reviews and for updated reviews, each in a version for searches of databases and registers only and a version that adds other sources.3 PRISMA 2020 replaces the 2009 statement, which its authors say should no longer be used, so a diagram drawn on the 2009 layout is itself an error.
Three units: record, report, study
PRISMA 2020 defines its terms, and the diagram cannot be completed correctly without them.1
| Unit | Meaning in PRISMA 2020 | Where it is counted |
|---|---|---|
| Record | The title or abstract, or both, of a report indexed in a database or website. Records that refer to the same report are duplicates | Identification, removal before screening, screening |
| Report | A document supplying information about a study: a journal article, preprint, conference abstract, register entry, clinical study report, dissertation or other document | Sought for retrieval, not retrieved, assessed for eligibility, excluded with reasons, reports of included studies |
| Study | An investigation, such as a clinical trial, with a defined group of participants and one or more interventions and outcomes. One study may have several reports | Studies included in review |
The unit changes twice on the way down the page. Screening is done on records. Full-text assessment is done on reports. Inclusion is counted in studies, with the number of reports alongside. A review that included 36 reports of 29 studies has to say both numbers, and its tables of study characteristics should have 29 rows.
The boxes and the arithmetic
The template for a new review that searched databases, registers and other sources has the following boxes.3
| Ref. | Box | Check |
|---|---|---|
| A | Records identified from databases and from registers | A = sum of the count for each database plus the count for each register |
| B | Records removed before screening: duplicates; marked ineligible by automation tools; removed for other reasons | B = sum of its three lines |
| C | Records screened | C = A minus B |
| D | Records excluded | From the screening log |
| E | Reports sought for retrieval | E = C minus D |
| F | Reports not retrieved | From the retrieval log |
| G | Reports assessed for eligibility | G = E minus F |
| H | Reports excluded, by reason | H = sum of the counts for each reason |
| I | Other methods: records identified from websites, organisations, citation searching and similar | I = sum of the count for each source |
| J, K | Other methods: reports sought for retrieval; reports not retrieved | J is at most I |
| L, M | Other methods: reports assessed for eligibility; reports excluded, by reason | L = J minus K; M = sum of its reasons |
| N | Reports of included studies | N = (G minus H) plus (L minus M) |
| S | Studies included in review | S is at most N; the difference is the number of additional reports of studies already counted |
The template carries two footnotes. The first asks authors to consider reporting the number of records from each database or register searched and not only the total. The second asks that, where automation tools were used, the diagram show how many records were excluded by a human and how many by the tool.3
Worked reconciliation
The numbers below are invented for a fictional review of randomised trials. They are illustrative only.
| Box | Count | Working |
|---|---|---|
| MEDLINE | 1,204 | |
| Embase | 1,587 | |
| CENTRAL | 412 | Databases: 3,203 |
| ClinicalTrials.gov | 86 | |
| WHO ICTRP | 61 | Registers: 147 |
| A. Records identified | 3,350 | 3,203 + 147 |
| B. Removed before screening (duplicates 1,012; automation tools 0; other 0) | 1,012 | |
| C. Records screened | 2,338 | 3,350 − 1,012 |
| D. Records excluded | 2,241 | |
| E. Reports sought for retrieval | 97 | 2,338 − 2,241 |
| F. Reports not retrieved | 5 | |
| G. Reports assessed for eligibility | 92 | 97 − 5 |
| H. Reports excluded: wrong population 22; wrong comparator 15; outcome not reported 11; not randomised 9; duplicate report 4 | 61 | 22 + 15 + 11 + 9 + 4 |
| Reports carried forward | 31 | 92 − 61 |
| Box | Count | Working |
|---|---|---|
| I. Records identified: citation searching 24; websites 6 | 30 | 24 + 6 |
| J. Reports sought for retrieval | 30 | |
| K. Reports not retrieved | 2 | |
| L. Reports assessed for eligibility | 28 | 30 − 2 |
| M. Reports excluded: wrong population 12; not randomised 8; outcome not reported 3 | 23 | 12 + 8 + 3 |
| Reports carried forward | 5 | 28 − 23 |
| N. Reports of included studies | 36 | 31 + 5 |
| S. Studies included in review | 29 | 7 reports were further publications of studies already counted |
Every line can be traced to a log: the export count from each database on the search date, the deduplication report from the reference manager or screening tool, the title and abstract decisions, the list of full texts requested, and the full-text decisions with one reason each. If any of those logs was not kept, the diagram cannot be rebuilt and the search has to be rerun to recover the counts.
Ten counting errors
- Database counts that do not sum to the total. The per-source numbers in the diagram, the text and the search appendix should be identical. Differences usually mean the search was rerun on another date and only one place was updated.
- Duplicates removed in the wrong place. Duplicate records come out before screening, in box B. If duplicates found later are added to the "records excluded" box, C minus D no longer equals E. Duplicate reports found at full text are a reason in box H, as in the example.
- Other sources folded into the database total. Records from citation searching, websites or contact with organisations belong in the right-hand column. Adding them to box A overstates the yield of the database search and breaks the left-hand arithmetic.
- The "not retrieved" box left out. Reports that could not be obtained are neither assessed nor excluded for a reason. Omitting box F forces them into H under an invented reason, or leaves E and G unequal with no explanation.
- Exclusion reasons that do not sum. Each excluded report gets one primary reason. Where a report fails several criteria, apply the criteria in a fixed order and record the first that fails. State that order in the Methods.
- A vague reason carrying most of the exclusions. "Did not meet inclusion criteria" is not a reason. The elaboration paper asks for the primary reasons, such as ineligible study design or ineligible population.2
- Studies and reports confused in the final box. Reporting 36 "studies" when 36 reports describe 29 studies inflates the evidence base and will not match the forest plot or the characteristics table.
- Automation left invisible. If a classifier or a screening tool's prioritisation removed records, the number goes in box B or is split out in box D, as the template footnote asks.3
- The wrong template. An updated review uses the template for updates, which adds a box for the studies included in the previous version. A new review that searched only databases and registers uses the single-column version and deletes nothing that applies.
- The diagram disagrees with the abstract. The number of included studies in the abstract, the Results, the diagram and every table should be the same number. If fewer studies contribute to a meta-analysis than are included in the review, give the number per synthesis in the text and on the forest plot.
What goes with the diagram
The diagram summarises a process that other checklist items document in full. Item 6 asks for every database, register, website, organisation and other source searched or consulted, with the date each was last searched. Item 7 asks for the full search strategy for all databases, registers and websites, including filters and limits. Item 8 asks how many reviewers screened each record and each report, whether they worked independently and whether automation tools were used. Item 16b asks authors to cite studies that might appear to meet the inclusion criteria but were excluded, and to explain why.1
Two additions make the counts verifiable. PRISMA-S, the extension for reporting literature searches, has 16 items covering each database and platform, full strategies, limits, dates and the method of deduplication.4 A supplementary table of reports excluded at full text, with citation and reason, satisfies item 16b and lets a reviewer confirm box H. The Cochrane Handbook (version 6.5) remains the methods reference for how searching and selection should be carried out.5
A reconciliation routine before submission
- Freeze the search
- Record the date and the export count for every source. If the search is updated before submission, add the new records as a separate run and report both dates.
- Keep the logs
- Deduplication report; title and abstract decisions; full-text request list with outcome; full-text decisions with one reason each; a study-to-report linkage table.
- Run the equations
- C = A − B; E = C − D; G = E − F; reports carried forward = G − H; the same for the right-hand column; N = sum of both columns; S from the linkage table.
- Cross-check
- Abstract, Results text, diagram, characteristics table, risk-of-bias table and forest plots all give the same number of studies.
A reconciled PRISMA flow diagram, the search strategies and the completed PRISMA 2020 checklist are deliverables of our systematic review and meta-analysis service. A review that is already drafted can go through medical manuscript editing. Both are set out under journal manuscripts.
Do I have to list the number of records from each database?
The template footnote asks authors to consider it where feasible, and the elaboration paper notes that per-database counts make it easier for others to check whether they have replicated a search. Give them in the diagram or in the search appendix.
Where do records from trial registers go?
In the left-hand column with databases, on their own line. PRISMA 2020 separates databases and registers from other methods such as websites, organisations and citation searching.
Can I remove boxes that do not apply?
Yes. The figure legend in the PRISMA 2020 statement says the grey boxes are completed only if applicable and otherwise removed. Do not remove a box that applies but has a count of zero; report the zero.
Is the PRISMA 2020 diagram used for scoping reviews?
Scoping reviews follow the PRISMA-ScR extension for their checklist. The results of searching and selection are still reported, and the PRISMA 2020 template can be adapted for the purpose.6 Check the journal's instruction.
References
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. PMID 33782057 · DOI
- Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. PMID 33781993 · DOI
- PRISMA. PRISMA 2020 flow diagram templates. prisma-statement.org/prisma-2020-flow-diagram
- Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst Rev. 2021;10(1):39. PMID 33499930 · DOI
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5 (updated August 2024). Cochrane; 2024. cochrane.org/handbook
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. PMID 30178033 · DOI