CalibrumStudy design guide
Practical Delphi Methodology GuideDelphi study design and reporting checklist
A rigorous Delphi study requires more than several rounds of questions. Use this checklist to plan the panel, feedback, consensus rules, analysis, and reporting decisions that make the process transparent, reproducible, and defensible.
Study designPanel representationConsensus criteriaControlled feedbackTransparent reporting
Before choosing Delphi
Begin by confirming that Delphi fits the research question. It is most useful when informed judgment must be gathered systematically, evidence is incomplete or contested, and a diverse panel needs a structured way to identify agreement and disagreement.
Confirm the purpose
- State the decision, definition, recommendation, priority, standard, or outcome the study must develop.
- Explain why existing evidence or ordinary survey methods are insufficient.
- Define the intended use of the final findings.
- Identify the steering committee, methodological lead, and final decision authority.
Choose the Delphi format
- Traditional multi-round, modified, Real-Time, RAND/UCLA, or a documented hybrid design.
- Number and purpose of planned rounds or stopping logic.
- Whether a meeting or live consensus phase is methodologically necessary.
- What must remain anonymous and what may be discussed openly.
1. Panel composition and stakeholder representation
Panel quality depends on who is invited, who participates, and whether the perspectives required by the research question are adequately represented—not simply on the total number of respondents.
Define eligibility
- Set explicit expertise or lived-experience criteria.
- Describe recruitment sources and selection procedures.
- Document exclusions, conflicts, and replacement rules.
Justify panel size
- Set a target based on the expertise and perspectives required.
- Allow for expected nonresponse and attrition across rounds.
- Explain why the planned size is adequate for the study and any subgroup review.
Map stakeholder groups
- Identify every perspective needed for a credible result.
- Set recruitment targets where representation matters.
- Consider geography, discipline, setting, role, and lived experience.
Monitor actual participation
- Track invited, started, completed, and retained participants.
- Review participation by stakeholder group and round.
- Assess whether attrition changes the composition of the panel.
There is no universal minimum panel size. Predefine and report a reasoned target; then report who actually contributed to each round and explain important gaps, imbalances, or changes in panel composition. See our panel-size guide.
2. Questionnaire and item development
Develop the content
- Describe the evidence review, interviews, workshops, prior framework, cognitive mapping, or open round used to generate items.
- Use one clear proposition or construct per item.
- Separate clarity, agreement, importance, feasibility, validity, and appropriateness when they require different judgments.
- Provide definitions and evidence summaries where needed.
Test the instrument
- Pilot with people resembling the intended panel.
- Review wording, scale interpretation, usability, and completion burden.
- Test instructions, feedback displays, branching, and access.
- Document substantive changes made after testing.
3. Rating scales and consensus rules
Consensus should be defined before results are reviewed. The criteria should match the scale, research purpose, and consequences of classifying an item as consensus, disagreement, or unresolved.
Specify the scale
- Define every response option and anchor.
- Explain any neutral, uncertain, unable-to-rate, or not-applicable option.
- Keep scale direction consistent.
Predefine the rules
- State numerical inclusion and exclusion thresholds.
- Define disagreement, near-consensus, and unresolved status.
- Specify subgroup rules if they affect the decision.
Define multidimensional decisions
- Set a separate rule for every required rating dimension.
- State whether an item must pass all criteria or whether some dimensions are descriptive.
- Report which criterion caused an item to be retained, revised, excluded, or carried forward.
Plan item and round decisions
- Determine what happens to included, excluded, revised, and unresolved items.
- Define when new items may enter and when another round is required.
- Predefine stopping conditions and document exceptions or post-launch decisions.
A single consensus percentage may be insufficient. An item can be highly important yet too unclear to retain, or valid but not feasible to implement. See how to plan multidimensional consensus criteria.
4. Rounds, feedback, and reconsideration
Plan controlled feedback
- Specify which statistics, distributions, comments, and subgroup results panelists will see.
- Protect confidentiality when presenting qualitative reasoning.
- Avoid feedback that pressures participants toward an artificial majority.
- Explain whether earlier ratings remain visible and editable.
Manage iteration
- Give participants a genuine opportunity to reconsider.
- Track response changes when the study design requires stability analysis.
- Define reminder, reopening, late-response, and partial-response rules.
- Record when and why items are revised between rounds.
Consensus is not the only useful outcome. Stable disagreement, uncertainty, insufficient evidence, and different stakeholder perspectives may be important findings and should not be hidden by a single overall percentage.
5. Analysis and subgroup interpretation
| Analysis area | Questions to address | Reporting caution |
|---|---|---|
| Consensus status | Which items met the predefined rule, failed it, or remained unresolved? | Do not change thresholds after seeing the results without clearly reporting the deviation. |
| Multiple rating dimensions | Did each item pass every required criterion, and which dimension caused a failure? | Do not combine clarity, importance, feasibility, validity, or appropriateness into one conclusion unless the protocol justifies it. |
| Distribution and disagreement | Are ratings tightly grouped, polarized, dispersed, or dominated by uncertain responses? | A median or overall percentage alone can conceal meaningful disagreement. |
| Stable polarization | Do two defensible response clusters persist after feedback and reconsideration? | Do not force one conclusion when the evidence supports distinct context-dependent positions. |
| Round-to-round movement | Did judgments converge, remain stable, or move in different directions? | Movement does not automatically mean improved validity. |
| Stakeholder groups | Do clinically or methodologically relevant groups show different patterns? | Overall consensus can conceal a group-specific objection; small subgroup counts are usually descriptive and should not be overinterpreted. |
| Qualitative reasoning | What explanations, concerns, suggested revisions, implementation barriers, or minority perspectives clarify the ratings? | Describe how comments were coded, summarized, and selected for feedback. |
| Participation | Who responded in each round, and did attrition alter panel composition? | Report denominators clearly for every round and analysis. |
6. What a transparent final report should include
Methodology
- Rationale and Delphi variant
- Panel eligibility and recruitment
- Stakeholder composition
- Questionnaire development and testing
- Rounds, rating dimensions, feedback, consensus rules, and stopping decisions
Results
- Invitations, participation, completion, and attrition
- Criterion-specific consensus, disagreement, and unresolved items
- Response distributions and round-to-round findings
- Stakeholder-group patterns where appropriate
- Qualitative themes, implementation barriers, and item revisions
Interpretation and reproducibility
- Final recommendations or agreed items
- Minority and divergent perspectives
- Protocol changes and deviations
- Limitations and representation gaps
- Questionnaire, decision rules, round history, data dictionary, and suitable anonymized supporting data
How Surveylet and Calibrum can help
Surveylet supports purpose-built Delphi workflows, including panel management, stakeholder grouping, multiple rounds, controlled feedback, Real-Time Delphi, consensus tracking, response history, and analysis-ready exports. Calibrum can also assist with study design, implementation, progress review, final analysis, and reporting.
Build methodological decisions into the study from the beginning
Tell us your research question, panel, intended outputs, and current study stage. Calibrum can help identify the design and reporting decisions that should be resolved before launch—or review an active study before the next round.