Can Experts Apply Your Delphi Recommendations Consistently?
Consensus on a recommendation is a useful result. Testing how people apply it to the same scenarios can reveal unclear definitions, different assumptions, and the questions that need further validation.
Agreement on a recommendation is one research question
A panel may agree with a recommendation while interpreting it differently when faced with the same case. A follow-up vignette exercise asks whether the resulting tool or guidance produces consistent decisions under defined conditions.
Decide which question the study needs to answer. Rating whether an action is appropriate for a case is part of developing recommendations. Asking people to apply an agreed tool to cases examines its interpretation and use. Both can be valuable, but they produce different evidence.
A vignette is a short, standardized scenario. It can describe a patient, service setting, professional task, or policy decision. Every respondent should receive the information necessary for the judgment being tested.
A recent study added case testing after consensus
An international medical-screening Delphi used a predefined 80% agreement rule before applying the resulting tool to 37 clinical vignettes. The authors reported Gwet agreement coefficients separately for professional and patient representatives: 0.86 and 0.76, respectively. They called for prospective multicenter validation before clinical implementation. Read the October 2026 study.
The additional stage examined how people applied the tool. It did not establish diagnostic accuracy or patient benefit. This distinction helps research teams describe what their Delphi findings support and what later testing must establish.
Design cases around the decisions you need to test
Freeze the version of the recommendation, definitions, and instructions before testing. Include typical cases, important boundary conditions, and situations where information is incomplete. Agree who reviews the scenarios for realism and unintended clues.
If the aim is to isolate a particular factor, change that factor while keeping other details consistent. If the aim is broader usability testing, sample a realistic range of cases and explain how they were selected. A convenient set of easy examples can conceal important ambiguity.
| Case type | Question to examine | Record |
|---|---|---|
| Typical case | Do respondents choose the same pathway? | Selected option and explanation. |
| Boundary case | Does one changed condition alter the decision consistently? | Changed factor, choice, and rationale. |
| Incomplete information | Do respondents recognize that a decision needs more information? | Missing-information response and requested detail. |
The examples illustrate a study structure, rather than clinical recommendations or results from the cited paper.
Collect independent responses before showing the answer pattern
If the objective is to assess independent application, collect the initial decisions before participants see group results or discuss the cases. Specify what training, reference material, and clarifications are available to everyone.
Feedback followed by repeat testing can be useful, but it answers an additional question about clarification or learning. Keep the initial test and later exercise separate. Record any changes to the tool, cases, or instructions.
Define which stakeholder groups perform which tasks. Professional and patient participants may contribute different forms of expertise; they do not automatically need identical clinical decision tasks or a pooled agreement estimate.
Choose the analysis for the response format
A percentage meeting a Delphi threshold and an interrater agreement coefficient describe different outcomes. Select an agreement measure appropriate to categorical, ordinal, or numerical answers, the number of raters, and the study design. Prespecify any weighting, missing-data handling, and confidence-interval method with suitable statistical input.
Inspect the case-level results as well as the overall estimate. A favorable average may hide disagreement on the cases with the greatest practical consequences. Retain explanations that identify unclear wording, missing assumptions, or different interpretations.
Agreement alone does not show that the decision is correct. Accuracy against an appropriate reference, feasibility in practice, and effects on outcomes require additional evidence where relevant.
Keep an application-testing record alongside the Delphi report
Report the version tested, case-development process, respondent groups, response instructions, feedback timing, and participation for each stage. Present the initial consensus results separately from the application-testing findings.
A useful export links each case identifier and version to the respondent or approved analysis identifier, stakeholder group, selected response, and explanation. Include the denominator for each case, exclusions, agreement analysis, and uncertainty. Explain how ambiguous cases or revisions affected the interpretation.
Surveylet can present scenarios, collect responses and comments, and export study data. Plan the case structure and required export with the research team; agreement coefficients can be calculated in an appropriate analysis workflow. Calibrum can help configure the exercise and agree the reporting scope.
Source and further reading
Ünlü L, Carpenter CR, Sterzer P, et al. International Delphi on a medical screening tool for emergency department patients with psychiatric complaints. BMC Medicine. Published online October 17, 2026; early accepted version.
For a study that uses cases to develop appropriateness judgments, see our RAND/UCLA case-design guide. The workflow and table above are planning suggestions for a separate application-testing stage.
Plan what happens after consensus
Bring the recommendation, the decisions it should support, and the cases you need to test. We can help organize the Surveylet exercise and define the data needed for analysis.