EditverseAnswering the statistical reviewer: a response-letter structure with examples
Six statistical comments that recur in peer review (multiplicity, missing data, model assumptions, sample size, confounding, subgroup claims), each with a model response and the manuscript change that goes with it.
Key points
- A statistical comment is answered with an analysis or a change to the text, and the response letter shows both. Argument without either seldom closes the point.
- Each response has four parts: the comment quoted in full, a direct answer, what was done, and the changed text with its location in the revised manuscript.
- Six comments recur: unadjusted multiple testing, unexplained missing data, unchecked model assumptions, absent or post hoc sample size reasoning, inadequate control of confounding, and subgroup claims based on separate P values.
- If a requested analysis changes the result, report the new result and change the conclusion. Reviewers check that the abstract moved with it.
- Label every analysis added at revision as post hoc, in the Methods and in the tables.
What the statistical reviewer is checking
Statistical review asks whether a reader with the data could reproduce the results and whether the conclusions follow from them. The ICMJE Recommendations (January 2026) put it this way for authors: describe the methods in enough detail for a knowledgeable reader with access to the data to judge their appropriateness and verify the results; quantify findings with measures of uncertainty such as confidence intervals; do not rely solely on P values; name the software and version; and distinguish prespecified from exploratory analyses, including subgroup analyses.1 Most statistical comments are a request to meet one of those five points.
The reporting guidelines turn them into items. CONSORT 2025 asks for the assumptions behind the sample size (item 16a), who was included in each analysis (21b), how missing data were handled (21c) and additional analyses marked as prespecified or post hoc (21d, 28).2 STROBE asks for the methods used to control confounding, examine subgroups, address missing data and test sensitivity (item 12).3
The structure of the response
| Part | Content | Why it is there |
|---|---|---|
| Comment | The reviewer's words, quoted in full and numbered (Reviewer 2, comment 3) | The editor reads the letter without the original report open |
| Answer | One sentence: agreed and done; agreed in part; or not done, with the reason | Tells the reviewer at once where the response is going |
| Action | The analysis run or the check made, with the method named and the result given as an estimate with its confidence interval | Shows the point was tested and not only discussed |
| Change | The new or revised text quoted, with section, paragraph and line in the tracked file; the table or figure number if one changed | Lets the reviewer verify the change without searching |
| Consequence | Whether the result or conclusion changed, and where else the manuscript was updated to match (abstract, discussion, limitations) | Prevents an abstract that contradicts the revised results |
Open the letter with a short summary of the main changes, especially any change to a primary result. Keep courtesy to a line. Where you disagree, say so in the answer line and give the methodological reason with a reference, then offer the requested analysis as a sensitivity analysis in the supplement so the reviewer can see what it shows.
Six recurring comments, with model responses
The examples below are illustrative. Studies, numbers and results are invented, and the wording is a model to adapt to the facts of the analysis.
1. Multiplicity
Typical comment: "The authors test fourteen secondary outcomes at P < 0.05 with no adjustment for multiple comparisons."
Adjustment is needed in a confirmatory setting when several tests feed one conclusion; it is not needed for analyses that are clearly presented as exploratory.4 Multiplicity arises from several endpoints, doses, populations and interim analyses, and each source has its own control strategies.5 The response therefore starts by sorting outcomes into those that support a claim and those that do not.
Illustrative response
We agree. The protocol named one primary outcome and three key secondary outcomes; the remaining ten were listed as exploratory. We have (a) applied the Holm procedure to the three key secondary outcomes, and (b) removed P values for the ten exploratory outcomes, which are now reported as estimates with 95% confidence intervals and described as exploratory. Unadjusted P values for the key secondary outcomes were 0.012, 0.030 and 0.21; the Holm-adjusted values are 0.036, 0.060 and 0.21. Only the first remains below 0.05. We have revised the Results (section 3.3, paragraph 2), Table 3 and the abstract, which no longer states that the intervention improved physical function.
The Holm working is worth showing: the smallest P value is multiplied by three, the next by two and the largest by one.
2. Missing data
Typical comment: "Nearly a fifth of participants lack the primary outcome. A complete-case analysis is presented with no justification."
No single method of handling missing data is valid in all circumstances. The assumptions behind the chosen method have to be stated and then tested in sensitivity analyses.6 For multiple imputation, reviewers expect the amount of missing data, the variables in the imputation model, the number of imputed datasets and a comparison with the complete-case result.7 In trials, ICH E9(R1) frames the question through the estimand: what happened to participants, what the trial is trying to estimate despite it, and sensitivity analyses that target the same estimand.8,9
Illustrative response
We agree that the original analysis was insufficient. The primary outcome was missing for 21 of 120 participants (17.5%) in the intervention group and 23 of 118 (19.5%) in the control group; reasons are now given by group in the flow diagram. We have replaced the complete-case analysis with multiple imputation under a missing-at-random assumption as the primary analysis (50 imputations; the imputation model included treatment group, baseline score, site, age and the week-6 score). The mean difference is now −1.8 points (95% CI −3.1 to −0.5), compared with −2.0 (95% CI −3.4 to −0.6) in the complete-case analysis, which is retained in supplementary table S4. We added a tipping-point analysis: the confidence interval would include zero only if missing outcomes in the intervention group were 2.5 points worse than imputed. Methods, section 2.7, and Results, section 3.2, have been rewritten.
3. Model assumptions
Typical comment: "A Cox model is reported, but there is no statement that the proportional hazards assumption was assessed."
General guidance on statistical reporting asks authors to state how the assumptions of each method were checked.10 The response gives the check, its result, and what was done where an assumption did not hold.
Illustrative response
We have added the assessment. Proportional hazards were examined with scaled Schoenfeld residuals and log-minus-log plots (supplementary figure S2). The global test gave P = 0.41. The assumption did not hold for tumour stage (P = 0.008), so the model is now stratified by stage. The hazard ratio for the exposure changed from 0.72 (95% CI 0.56 to 0.93) to 0.74 (95% CI 0.57 to 0.96). We also modelled age with a restricted cubic spline in place of a linear term; the estimate was unchanged to two decimal places. These checks are described in Methods, section 2.6.
4. Sample size and power
Typical comments: "No sample size justification is given." Or, after a null result: "Please report the post hoc power of the study."
The first request is met by reporting the original calculation with every assumption, or by stating that none was made and why the sample is what it is. The second is better answered by declining with a reason. A power figure computed from the observed effect restates the P value and adds no information, and misreadings of power and of non-significant results are among the errors catalogued in the standard guide to interpreting tests and intervals.11 The confidence interval already shows which effect sizes the data are compatible with.
Illustrative response
We have added the sample size calculation from the protocol to Methods, section 2.5: 84 participants per group gave 80% power to detect a difference of 2.0 points (the minimal important difference), assuming a standard deviation of 4.6 and two-sided alpha of 0.05; we enrolled 190 to allow for about 10% attrition. We have not calculated post hoc power, because power based on the observed effect is a function of the P value already reported. We agree that our original wording overstated the finding. The observed difference was 1.2 points (95% CI −0.2 to 2.6). Because this interval includes both zero and the minimal important difference, we now describe the result as inconclusive and have removed the statement that the treatments were "equally effective" from the abstract and the Discussion.
5. Confounding
Typical comment: "Covariates were selected by stepwise regression. Disease severity, a likely confounder, is not in the model."
STROBE asks authors to make clear which confounders were adjusted for and why they were included, and to give unadjusted and adjusted estimates side by side.3 A response that only defends the original model does not meet that. Refit the model with a covariate set chosen on subject-matter grounds, report what changed, and accept the result.
Illustrative response
We agree and have changed the approach. Covariates are now chosen on clinical grounds, before modelling, as common causes of exposure and outcome: age, sex, diabetes, renal function, baseline severity score and admission year. The rationale for each is in supplementary table S2. Stepwise selection is no longer used. With severity included, the adjusted odds ratio is 1.34 (95% CI 0.96 to 1.87); the unadjusted estimate is 1.82 (95% CI 1.35 to 2.45) and the previously reported adjusted estimate was 1.61 (95% CI 1.17 to 2.21). Table 2 shows all three. Because the interval now includes 1, we have revised the abstract, the first paragraph of the Discussion and the conclusion, which state that the association was attenuated after adjustment for severity and that residual confounding cannot be excluded.
6. Subgroup claims
Typical comment: "The authors conclude that the treatment works in women but not in men. This rests on a significant P value in one subgroup and a non-significant one in the other."
A difference between subgroups is assessed with a test of interaction, not by comparing P values calculated within each subgroup. Guidance on subgroup reporting in trials also asks for the number of subgroup analyses performed and which of them were prespecified.12
Illustrative response
We agree that the claim was not supported. We have tested the treatment-by-sex interaction: the ratio of hazard ratios is 0.81 (95% CI 0.52 to 1.25; P for interaction = 0.34). The subgroup estimates, with confidence intervals, are shown in a new forest plot (Figure 3) for the six prespecified subgroups; two post hoc subgroups added during analysis are shown separately and identified as post hoc. Methods, section 2.8, now states the number of subgroup analyses and that they were not adjusted for multiplicity. The sentence concluding a sex-specific effect has been removed from the abstract and the Discussion, which now say that there was no evidence that the effect differed by sex.
When you do not make the change
Some requests should be declined: an analysis the data cannot support, an adjustment for a variable on the causal pathway, a test that answers a different question. Decline in the answer line, give the reason in two or three sentences with a methodological reference, and where possible run the analysis anyway and place it in the supplement with a note on how to read it.
Before sending the letter
- Every comment answered
- Numbered in the reviewer's order, none merged or skipped, including comments from the editor.
- Numbers agree
- Each estimate quoted in the letter is identical to the one in the revised table and text.
- New analyses labelled
- Analyses added at revision are marked post hoc in the Methods and in table footnotes, as the ICMJE Recommendations and CONSORT 2025 ask for prespecified and exploratory analyses to be distinguished.
- Files consistent
- Tracked and clean versions match; supplementary tables are renumbered; the reporting checklist is updated for moved or added items.
Our reviewer response and revision service covers the response letter, the tracked and clean revised manuscript and a change log, with re-analysis and revised tables and figures in the Premium tier. Where the open question is the analysis itself, statistical analysis for clinical studies delivers the analysis report, tables, code and a methods paragraph. The wider service is described under statistical interpretation.
The reviewer asked for an analysis that changes our main result. Do we have to report it?
If the analysis is the more appropriate one, yes, and the conclusion should follow it. If you believe the original analysis is correct, keep it as primary, give the reason, and report the requested analysis as a sensitivity analysis with its result in full.
Should we adjust every P value in the paper for multiple testing?
Not necessarily. Adjustment is required where several tests support one confirmatory conclusion. Outcomes that are exploratory can be reported as estimates with confidence intervals, without significance claims, and described as exploratory.
Is it acceptable to refuse a request for post hoc power?
Yes, with a reason. Explain that power calculated from the observed effect is determined by the P value, and direct the reviewer to the confidence interval and to the original sample size calculation, both of which should be in the manuscript.
References
- International Committee of Medical Journal Editors. Preparing a manuscript for submission to a medical journal. In: Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. Updated January 2026. icmje.org
- Hopewell S, Chan AW, Collins GS, Hróbjartsson A, Moher D, Schulz KF, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ. 2025;389:e081123. PMID 40228833 · DOI
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453-7. PMID 18064739 · DOI
- Bender R, Lange S. Adjusting for multiple testing: when and how? J Clin Epidemiol. 2001;54(4):343-9. PMID 11297884 · DOI
- Dmitrienko A, D'Agostino RB Sr. Multiplicity considerations in clinical trials. N Engl J Med. 2018;378(22):2115-2122. PMID 29847757 · DOI
- Little RJ, D'Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012;367(14):1355-60. PMID 23034025 · DOI
- Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. PMID 19564179 · DOI
- International Council for Harmonisation. ICH E9(R1): Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. Step 4, 2019. database.ich.org
- Kahan BC, Hindley J, Edwards M, Cro S, Morris TP. The estimands framework: a primer on the ICH E9(R1) addendum. BMJ. 2024;384:e076316. PMID 38262663 · DOI
- Lang TA, Altman DG. Basic statistical reporting for articles published in biomedical journals: the "Statistical Analyses and Methods in the Published Literature" or the SAMPL guidelines. Int J Nurs Stud. 2015;52(1):5-9. PMID 25441757 · DOI
- Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337-50. PMID 27209009 · DOI
- Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007;357(21):2189-94. PMID 18032770 · DOI