Interpreting Statistical Significance in Delphi Studies
A significant agreement test does not establish that an item met your consensus rule or that judgments became more stable. Define each claim, retain comparable data, and report the evidence needed for that claim.
An agreement test can be significant in both rounds
A November 20, 2026 study of fitness-facility indicators reports that Kendall’s W for five main dimensions fell from 0.494 to 0.405 across two Delphi rounds. Both tests had p-values below 0.001. The authors explicitly avoid treating significance as evidence of increasing consensus.
The detailed indicator set also changed from 38 to 33 after deletions and additions. This makes the paper a useful example of two reporting tasks: interpreting an agreement statistic and establishing whether the material being compared stayed comparable.
The worked example below is fictional and uses Calibrum’s own numbers. It illustrates a reporting decision; it does not reproduce the study’s analysis or evaluate fitness facilities.
Define the question before selecting the statistic
| Question | Useful evidence | Interpretation |
|---|---|---|
| Did this item reach consensus? | The approved item rule, valid ratings, and response distribution. | Apply the protocol’s threshold and any additional conditions. |
| Do panelists rank these items similarly? | A suitable concordance measure, such as Kendall’s W, with its test assumptions. | Describe agreement in relative ordering across the rated set. |
| Did judgments become more stable? | Comparable wording, scales, participants, and paired ratings where available. | Assess change under the study’s planned stability definition. |
| Can the recommendation be used successfully? | An application test or implementation evaluation. | Explain what was tested beyond expert agreement. |
Kendall’s W summarizes concordance in rankings. Its significance test concerns evidence against a specified null hypothesis. The p-value does not provide the proportion supporting an individual statement, and W does not supply that statement’s consensus classification.
There is no universal W cutoff that establishes every Delphi study’s success. Choose an analysis appropriate to the rating task and document assumptions, ties, missing data, and the set of items included.
Track what changed before claiming progress
Imagine 20 panelists reviewing an administrative handoff process. The fictional protocol requires at least 80% of valid ratings in categories 4 or 5 on a five-point scale. All 20 panelists provide valid ratings for each item presented.
| Item history | Round 1 support | Round 2 support | What the record supports |
|---|---|---|---|
| H01: Unchanged receipt-confirmation statement. | 15/20 (75%). | 17/20 (85%). | Crossed the 80% item threshold under unchanged wording. |
| H02: Deadline changed from two days to five days. | 13/20 (65%). | 18/20 (90%). | The revised statement met the threshold; this is not a comparison of identical judgments. |
| H03: Escalation statement added after round 1. | Not presented. | 16/20 (80%). | Met the threshold at its first assessment; no prior rating exists. |
For H01, the net gain is two supporting ratings. The total number of people who changed their answer still requires linked individual data: movement in opposite directions can cancel out. Ratings may also move within a supporting category.
No W or p-value is calculated from this table. A concordance analysis requires the underlying respondent-by-item ratings and a defined set of comparable items.
Keep the comparison record alongside the result
Preserve a stable item identifier, wording version, dimension, scale, round, and valid count. Mark new, revised, retired, and unchanged items explicitly. A rising percentage of accepted items may partly reflect removal of difficult statements.
Document participation as well. All-respondent summaries and a matched-panel analysis answer different questions when panelists leave, return, or skip items. An apparent increase in support can reflect who responded.
If a rank-based measure falls while ratings remain high, inspect the full distributions, ties, and item changes before explaining the pattern. Report the observed values and their limits. Do not infer a cause from the two coefficients alone.
For Real-Time Delphi, define the observation times and feedback opportunities used in a comparison. Repeated snapshots of unchanged data provide a record of the same state, with no additional evidence that panelists reconsidered their judgments.
Write the claim the data can support
Original reporting example
“H01 met the prespecified 80% rule in round 2 (17/20, 85%), following 15/20 (75%) in round 1. Wording and scale were unchanged. Individual rating stability was not assessed.”
Surveylet provides ratings, feedback, and exports for the agreed analysis. Specify any Kendall’s W, matched-rating comparison, or custom reporting table as an analysis requirement. A displayed consensus indicator should be interpreted under its configured rule.
For the broader comparison plan, see what changed between Delphi rounds. When a protocol requires repeated threshold attainment, see consecutive-round consensus.
Sources and scope
Construction and empirical test of an evaluation system for the high-quality development of national fitness facilities in Hunan Province. Frontiers in Public Health. Published November 20, 2026. DOI: 10.3389/fpubh.2026.1970856. See the Delphi consultation results and Table 3.
The ACCORD reporting guideline supports transparent description of consensus methods. The handoff scenario, comparison table, and reporting sentence are original Calibrum illustrations. The cited research is used as a methodology example; it is not presented as a Surveylet customer study.
Plan the analysis before the first round
Share your protocol, planned rounds, and reporting requirements so we can scope the Surveylet configuration and analysis support.
