How to Plan a Delphi Study
A practical guide to planning expert panels, rounds, rating scales, feedback, consensus thresholds, and reporting before launching a Delphi or Real-Time Delphi study.
Good Delphi studies begin before the first survey is launched
A Delphi study is more than a questionnaire. It is a structured process for gathering expert judgment, providing feedback, and identifying areas of agreement, disagreement, or priority. Because the process is iterative, planning decisions made at the beginning can strongly affect the quality and credibility of the final results.
Before launching a Delphi study, the research team should clarify the study objective, define the expert panel, choose the number of rounds, select rating scales, decide how feedback will be shown, and determine how consensus will be measured.
Careful planning helps the study run more smoothly and makes the final results easier to interpret, defend, and report.
Step 1: Define the purpose of the study
The first planning question is simple but important: what should the Delphi study produce? A vague goal such as “get expert input” is usually not enough. The team should define the expected output as clearly as possible.
Consensus recommendations
Statements, guidelines, standards, care pathways, policy options, or decision criteria that require expert agreement.
Prioritized items
Research questions, risks, outcomes, competencies, barriers, or interventions ranked by importance or feasibility.
Refined definitions
Shared terminology, eligibility criteria, measurement definitions, or conceptual frameworks.
Step 2: Decide whether Delphi is the right method
Delphi is useful when expert judgment matters and when the study needs more than one-time feedback. It may not be necessary for every survey or consultation process.
| Delphi may be a good fit when... | A simpler survey may be enough when... |
|---|---|
| The goal is expert consensus, prioritization, recommendations, or agreed definitions. | The goal is to collect one-time opinions, experiences, or satisfaction ratings. |
| The topic is complex, uncertain, emerging, or dependent on professional judgment. | The topic is straightforward and does not require structured reflection. |
| Participants should review feedback and have an opportunity to reconsider responses. | Participants only need to answer once. |
| The results need transparent consensus criteria and a clear methodological record. | The results only require descriptive summaries. |
Step 3: Define the expert panel
The credibility of a Delphi study depends heavily on who participates. The panel should be selected based on the study objective, not simply convenience.
Expertise
Participants may be selected because of clinical, research, policy, technical, educational, or operational expertise.
Stakeholder role
Some studies include patients, caregivers, administrators, community leaders, professional societies, or decision-makers.
Representation
The team should consider disciplines, geography, institution type, professional background, and lived experience where relevant.
The panel size can vary widely depending on the topic, available experts, and stakeholder groups. More participants are not always better if the group is poorly defined. A smaller, well-selected panel may be more useful than a large group with unclear relevance.
Step 4: Choose the Delphi format
Delphi studies can be designed in different ways. The best format depends on the study objective, timeline, topic, panel availability, and need for feedback between rounds.
Traditional multi-round Delphi
Participants complete separate rounds. The team summarizes results between rounds and may revise items, remove items, add items, or provide structured feedback before the next round.
- Good for formal round-by-round documentation
- Useful when feedback requires manual review
- Helpful when items may change substantially between rounds
Real-Time Delphi
Participants may see updated group feedback while the study is still active and revise their answers during the same study window.
- Good for faster timelines
- Useful for geographically dispersed or busy panels
- Helpful when feedback can be shown dynamically
Step 5: Develop clear items and questions
Delphi studies often ask participants to rate, rank, comment on, or revise a set of items. These items might be recommendations, outcomes, competencies, risks, definitions, scenarios, or research priorities.
Poorly written items can create confusion and weaken the results. Each item should be understandable, specific, and aligned with the study objective.
Use precise wording
Avoid vague, double-barreled, overly broad, or leading statements.
Separate ideas
If an item contains two concepts, participants may agree with one part and disagree with the other.
Include comment prompts
Comments can explain disagreement, suggest revisions, or identify missing items.
Step 6: Select rating scales
Rating scales should match what the study is trying to measure. A Delphi study may ask participants to rate importance, feasibility, appropriateness, agreement, urgency, relevance, clarity, or other dimensions.
Importance
Useful for ranking outcomes, research priorities, competencies, risks, or topics.
Agreement
Useful for evaluating statements, recommendations, definitions, or standards.
Appropriateness
Useful for clinical scenarios, decision criteria, policy choices, or intervention judgments.
Many Delphi studies use Likert-type scales, such as 1–5, 1–7, or 1–9 scales. The scale should be clearly labeled so participants understand what each number means.
Step 7: Define consensus criteria before launch
A Delphi study should define how consensus will be measured before the results are interpreted. This avoids the appearance that thresholds were chosen after seeing the data.
Percentage agreement
For example, the percentage of participants rating an item above a certain threshold.
Median rating
Useful when the study uses numerical scales and wants a central tendency measure.
Interquartile range
Helpful for assessing how tightly responses cluster around the median.
Stability across rounds
Shows whether responses are still shifting or whether the panel’s views have stabilized.
Ranking thresholds
Useful when the output is a prioritized list rather than simple inclusion or exclusion.
Stakeholder agreement
Some studies require consensus overall and within specific stakeholder groups.
Step 8: Plan feedback between rounds
Feedback is one of the features that separates Delphi studies from traditional surveys. Participants should receive enough information to reflect on their responses without being pressured to conform.
Summary statistics
Median ratings, distributions, percentage agreement, or consensus status can help participants understand the group view.
Anonymized comments
Selected comments can explain why participants rated an item differently or suggest wording improvements.
Individual comparison
Some studies show participants how their prior response compares with the group summary.
The study team should decide what feedback participants will see, when they will see it, and how it will be used to support revision in the next round or during a Real-Time Delphi workflow.
Step 9: Prepare the administration workflow
Even a well-designed Delphi study can struggle if the operational workflow is not ready. The team should plan how participants will be invited, reminded, supported, and tracked.
Before launch
- Finalize the participant list and contact information.
- Prepare invitation and reminder text.
- Test survey access, consent language, and instructions.
- Confirm whether responses are anonymous or identifiable to the study team.
- Review the data export and reporting plan.
During the study
- Monitor response rates.
- Send reminders at planned intervals.
- Track technical or participant questions.
- Prepare feedback between rounds if needed.
- Document changes to items, wording, or workflow.
Step 10: Plan analysis and reporting
The final report should explain not only what the panel agreed on, but also how the study was conducted. A strong report makes the process transparent and helps readers understand the credibility of the results.
Describe the panel
Summarize participant groups, relevant expertise, stakeholder roles, geography, or professional backgrounds.
Report the process
Describe the number of rounds, feedback provided, response rates, item changes, and consensus criteria.
Present the results
Show final included items, excluded items, rankings, areas of disagreement, and changes across rounds.
Common planning mistakes to avoid
Many Delphi problems can be prevented with careful planning. The most common issues are usually not technical; they are methodological or operational.
Unclear study objective
If the team does not know what the study should produce, the questionnaire and analysis plan may drift.
Weak panel definition
Unclear inclusion criteria can make it difficult to defend the relevance of the expert panel.
Undefined consensus rules
Consensus thresholds should be planned before results are interpreted.
Poor item wording
Ambiguous or double-barreled items can create confusion and weaken the findings.
Too much participant burden
Long surveys, unclear instructions, or too many rounds can reduce participation and data quality.
Insufficient reporting plan
The team should know how results will be summarized, exported, and presented before the study begins.
How Surveylet supports Delphi study planning and administration
Surveylet was designed to support Delphi and Real-Time Delphi studies, including expert panels, structured questionnaires, multiple rounds, controlled feedback, response tracking, and consensus-oriented exports.
Calibrum can also support teams that need help planning the study, refining questions, setting up Surveylet, administering rounds, analyzing results, and preparing reporting materials.
Platform support
Use Surveylet to manage Delphi and Real-Time Delphi workflows online.
Study setup support
Get help structuring the survey, rounds, feedback, instructions, and participant workflow.
Reporting support
Export and summarize results for steering committees, reports, manuscripts, and supplemental materials.
Planning a Delphi study?
Surveylet helps researchers and organizations manage expert panels, structured feedback, multiple rounds, Real-Time Delphi workflows, response tracking, and consensus-oriented reporting.