Rate First, Then Shortlist: How to Prioritize in a Delphi Study
A panel can agree that a topic belongs in a program and still give other topics priority for the first phase. Use one question to assess importance and a separate ballot to decide what to tackle first.
A curriculum study separated importance from priority
In a November 2026 Delphi study on intensive care recovery education, van Mol and colleagues collected nine-point importance ratings and asked participants to choose up to ten priorities. Each selection counted as one vote, and the authors ranked topics by their selection totals.
A subsequent meeting used the results and qualitative themes to assemble curriculum modules, without a new ballot. The researchers conducted their surveys in SurveyMonkey. The volunteer training example below is an original Calibrum illustration of the method.
Decide what each stage will establish
Importance: Define the purpose of the program, the rating bands, who can assess each item, the minimum number of valid ratings, and the consensus rule. Give panelists an explicit response for topics they cannot assess and say how it affects the denominator.
Priority: Specify which topics go on the ballot, how many each panelist can select, and how votes are counted. If the ballot is meant to sequence topics that already qualify for inclusion, publish that eligibility rule in advance. If its purpose is broader exploration, label the resulting priorities accordingly.
A cap of five selections per person and a limit of five topics in the first phase are separate decisions. If there are five places and two topics tie for the last one, apply the planned tie rule or report the place as unresolved. Attribute any later editorial or implementation decision to the group that made it.
Adapt these two questions to your topic list
For a fictional volunteer orientation program, the rating list contains ten topics: V01 Role boundaries; V02 Protecting personal information; V03 Using the scheduling tool; V04 Requesting help during a shift; V05 Accessibility support; V06 Finding key locations; V07 Supporting new volunteers; V08 Recording shift notes; V09 Using shared equipment; and V10 Responding to a missed handoff.
Stage 1: Rate each topic’s importance
“How important is it to include this topic in the volunteer orientation program?”
Use 1–3 for not important, 4–6 for important but not essential, and 7–9 for essential. Offer “Unable to assess” separately. In this example, the agreed rule is at least 70% of valid ratings in 7–9 and no more than 15% in 1–3, subject to the study’s minimum number of valid ratings.
Stage 2: Select up to five topics to develop first
“Which eligible topics should the program develop first? Select up to five. Each selected topic counts as one vote; you do not need to order your choices.”
For this example, show only topics that met the inclusion rule. Use the same topic labels and IDs. Specify how the study will distinguish a deliberate decision to select none from a ballot that was never submitted.
Give ratings and ballots their own denominators
Three rows from the fictional study are shown below. Each has 20 valid importance ratings. Eighteen of the 20 panelists submitted a valid priority ballot. The other two did not submit a ballot.
| Topic | Rated 7–9 | Rated 1–3 | Inclusion rule | Priority selections |
|---|---|---|---|---|
| V01 Role boundaries | 17/20 (85%) | 1/20 (5%) | Met | 6/18 (33.3%) |
| V04 Requesting help during a shift | 15/20 (75%) | 2/20 (10%) | Met | 14/18 (77.8%) |
| V09 Using shared equipment | 14/20 (70%) | 4/20 (20%) | Did not meet | Not on the ballot |
V01 has the higher importance rating share, but V04 receives more votes to develop it first. V09 exceeds the low-rating limit and did not appear on the ballot; its priority share is therefore not applicable. Percentages across a ballot can add to more than 100% because each person can select multiple topics.
Report each stage’s eligibility, response counts, missing answers, rule, and outcome. If a group later combines topics into modules or decides what can be delivered first, record that decision and its rationale separately from the vote.
Choose the order based on the claim you need to make
Rate first when you want importance judgments from the whole eligible panel before asking it to set priorities. Set expectations about workload and keep the two tasks clearly labeled.
Choosing first and rating only the selected items can reduce workload for a screening exercise. Each item’s ratings then describe the people who selected and rated it. Keep that base visible, and keep items a panelist was never shown separate from missing answers or low ratings. See reporting preferences and abstentions for more on response formats and denominators.
Rehearse the questionnaire and the export in Surveylet
Use a rating scale or matrix for importance and a separate multiple-selection question for the ballot. The Surveylet question types guide explains checkbox and ranking formats. Choose checkboxes when each selection counts equally.
Verify that your configured ballot handles the permitted number of choices, attempts to exceed the cap, intentional empty ballots, and missing answers as the protocol specifies. If the setup does not enforce the desired limit, agree how to implement and validate it before launch. Test the exports to confirm both stages can be joined by topic ID and their denominators can be reproduced.
Test threshold boundaries with known counts. Three low ratings out of 20 are exactly 15%. The example and published study include that boundary; the preset described in the consensus rules guide uses less than 15% for the low band. Where the selected configuration differs from your protocol, apply and document the approved classification using the exported counts.
Source and scope
van Mol M, Karner V, Akerman E, et al. Establishing curriculum priorities for intensive care recovery education: an international Delphi study. Intensive Care Medicine. Published November 2, 2026.
The article provides the methodological precedent. The sample questions, five-choice cap, and results table are original examples for planning and testing a Surveylet study.
Plan the questions and the report together
Decide how to assess importance, set priorities, and explain the result before opening your study.