Common Delphi Study Mistakes
Practical issues that can weaken Delphi studies — and how research teams can avoid them through better planning, clearer methodology, stronger feedback design, and purpose-built workflows.
Delphi studies are powerful, but they need careful design
The Delphi method can be an excellent way to gather expert input, refine ideas, and identify consensus around complex questions. It is used in medical research, policy, education, technology, planning, and many other fields where expert judgment matters.
But a Delphi study is not automatically rigorous just because it uses multiple rounds or expert participants. The quality of the study depends on clear objectives, appropriate panel selection, well-written items, meaningful feedback, predefined consensus rules, and transparent reporting.
Many Delphi problems can be prevented before the study begins. The most common mistakes are usually methodological, not technical — although the wrong software workflow can make those problems harder to manage.
Mistake 1: Starting without a clear study objective
A vague objective is one of the most common causes of Delphi study problems. If the team is not clear about what the study should produce, it becomes difficult to select the right panel, write good items, choose rating scales, define consensus, or interpret the results.
Weak objective
“We want to get expert opinions about this topic.”
Stronger objective
“We want to identify consensus recommendations for managing this condition in settings where evidence is limited.”
Mistake 2: Treating Delphi as a normal survey
Delphi studies often use questionnaires, but the method is more than a questionnaire. A traditional survey collects responses. A Delphi study uses structured feedback, reflection, and consensus criteria to help participants reconsider and refine their views.
If a study only asks experts to answer once and then reports the results, it may be a useful expert survey, but it may not truly function as a Delphi study.
| Traditional survey thinking | Delphi study thinking |
|---|---|
| Collect responses once and summarize them. | Collect responses, provide feedback, and allow reconsideration. |
| Focus on descriptive results. | Focus on structured consensus, priorities, or recommendations. |
| Use respondents as a sample. | Select participants because of expertise or stakeholder relevance. |
| Report what people answered. | Report how agreement was developed, measured, and interpreted. |
Mistake 3: Poorly defining the expert panel
The strength of a Delphi study depends on the relevance and credibility of the panel. If the participant criteria are unclear, readers may question whether the panel was appropriate for the study objective.
Unclear expertise
The team does not explain why participants were qualified to contribute.
Missing stakeholders
Important perspectives, such as patients, practitioners, administrators, or policy experts, may be excluded.
Unbalanced representation
One discipline, region, organization, or viewpoint may dominate the panel unintentionally.
Mistake 4: Using unclear or double-barreled items
Delphi participants can only rate what they understand. If items are vague, too broad, leading, or contain multiple ideas, participants may interpret them differently. This can create disagreement that reflects wording problems rather than true differences in expert judgment.
Problem item
“The intervention should be effective, affordable, and easy to implement.”
- Effectiveness, affordability, and implementation are different concepts.
- A participant may agree with one part but disagree with another.
Better approach
Separate the concepts into distinct items or rating dimensions.
- Effectiveness
- Affordability
- Ease of implementation
Mistake 5: Choosing the wrong rating scale
Rating scales shape how participants interpret the task. A scale that does not match the study objective can make results difficult to interpret. For example, rating “importance” is different from rating “agreement,” “feasibility,” “appropriateness,” or “urgency.”
Importance
Useful for outcomes, research priorities, competencies, risks, or topics.
Agreement
Useful for statements, recommendations, definitions, or standards.
Appropriateness
Useful for clinical scenarios, interventions, decisions, or criteria.
Mistake 6: Defining consensus after seeing the results
Consensus criteria should be defined before results are interpreted. If thresholds are selected after seeing the data, the study may appear less objective or less credible.
Percentage agreement
For example, the percentage of participants rating an item above a predefined level.
Median or distribution
Useful when the study uses numerical ratings and wants to summarize the center and spread of responses.
Stability across rounds
Helpful for assessing whether opinions are still changing or have become stable.
Mistake 7: Providing weak or confusing feedback between rounds
Feedback is one of the defining features of Delphi. If feedback is missing, unclear, overly detailed, or too persuasive, participants may not be able to use it effectively.
Good feedback helps participants understand the group’s responses while preserving independent judgment. It should support reflection, not pressure participants to conform.
Helpful feedback
Summary statistics, distributions, prior response comparison, consensus status, and selected anonymized comments.
Problem feedback
Unclear summaries, excessive raw comments, unexplained charts, or language that implies participants should change.
Participant guidance
Explain what the feedback means and how participants should use it when reconsidering their responses.
Mistake 8: Making the study too long or burdensome
Delphi studies often involve busy experts. If the survey is too long, the instructions are confusing, or the number of rounds is excessive, response rates and data quality can suffer.
Common burden problems
- Too many items in a single round
- Unclear instructions
- Repetitive questions
- Too many open-ended prompts
- Unrealistic deadlines
- Too many rounds without a clear reason
Better design choices
- Prioritize the most important items
- Use concise instructions
- Group items logically
- Use comments where they add value
- Set realistic timelines
- Explain why participation matters
Mistake 9: Not planning for analysis and reporting
Some teams focus heavily on launching the survey but do not plan how results will be analyzed, summarized, and reported. This can create problems when it is time to prepare manuscripts, steering-committee summaries, or final recommendations.
Before launch
Define the analysis plan, consensus criteria, subgroup comparisons, and reporting outputs.
During the study
Track response rates, item changes, feedback decisions, and comments that may need review.
After completion
Prepare tables, figures, consensus classifications, narrative summaries, and supplemental materials.
Mistake 10: Using tools that do not support Delphi workflows
General-purpose survey tools can be useful for standard questionnaires, but Delphi studies often require workflows that go beyond basic data collection. Study teams may need to manage rounds, carry items forward, show feedback, track response changes, compare stakeholder groups, and export analysis-ready data.
If the software does not support the Delphi process, the team may need to perform more manual work, increasing the risk of delays, inconsistent feedback, data-management problems, or reporting errors.
Round management
Support separate rounds, reminders, response tracking, and participant workflows.
Structured feedback
Present group results, comments, prior responses, or consensus status in a controlled way.
Reporting exports
Export data for consensus analysis, steering committees, manuscripts, and supplemental materials.
A simple Delphi planning checklist
Before launching a Delphi study, the study team should be able to answer the following questions clearly.
Methodology checklist
- What is the study objective?
- What final output should the study produce?
- Who should be on the expert or stakeholder panel?
- How many rounds are expected?
- What rating scales will be used?
- How will consensus be defined?
Workflow checklist
- How will participants be invited and reminded?
- What feedback will participants see?
- Will comments be collected and summarized?
- How will items be revised or carried forward?
- How will data be exported and analyzed?
- How will results be reported?
How Surveylet helps reduce common Delphi study problems
Surveylet was designed to support Delphi and Real-Time Delphi studies, including expert panels, structured questionnaires, multiple rounds, feedback, response tracking, and consensus-oriented exports.
Calibrum can also support teams that need help planning the study, refining items, setting consensus rules, configuring Surveylet, administering rounds, analyzing results, and preparing reporting materials.
Better structure
Support multi-round and Real-Time Delphi workflows that match the study design.
Better feedback
Help participants review group results and reflect before revising responses.
Better reporting
Export and summarize results for research teams, steering committees, manuscripts, and final reports.
Want to avoid common Delphi study pitfalls?
Surveylet helps researchers and organizations manage expert panels, structured feedback, multiple rounds, Real-Time Delphi workflows, response tracking, and consensus-oriented reporting.