Delphi Quorum and Agreement: Two Checks Before Accepting an Item
A high agreement percentage can come from a small part of the eligible panel. A participation requirement establishes whether enough people rated the item; an agreement requirement establishes how much support it received among the ratings that count.
Keep participation and support in separate fields
For a study using a participation threshold, calculate item coverage as substantive ratings divided by eligible panelists. Calculate agreement using the supporting ratings and the denominator specified in the protocol. If acceptance requires both thresholds, meeting one is insufficient.
Define a substantive rating as a response on the item’s rating scale. Keep explicit abstentions and missing answers visible in separate counts. A midpoint response still counts as a rating unless the approved analysis plan explicitly specifies otherwise.
The term “quorum” can have different meanings across organizations. Define whether the requirement applies to each item, a round, a meeting, or a stakeholder group. Meeting attendance alone cannot establish how many people rated a particular item.
ISHLT gives a concrete example of separate thresholds
The ISHLT Delphi guidance for authors specifies a five-point agreement scale, at least 80% participation per statement, and at least 75% agreement among voting panelists. Ratings of 4 or 5 count toward agreement; abstentions are excluded from that agreement calculation. The guidance also calls for ACCORD reporting.
These are requirements for that organization’s consensus documents. A different study needs its own justified rules. The document’s publication date is not stated; this page uses the guidance accessed October 23, 2026.
The same agreement percentage can support different decisions
The following original examples use 25 eligible panelists and illustrative requirements of at least 80% substantive item ratings and at least 75% agreement among those ratings. An explicit non-rating contributes to the abstention count, while a blank contributes to missingness.
| Item | Ratings / eligible | Abstentions / missing | Agreement | Decision |
|---|---|---|---|---|
| A | 15/25 = 60% | 6 / 4 | 12/15 = 80% | Insufficient participation. |
| B | 20/25 = 80% | 3 / 2 | 15/20 = 75% | Meets both thresholds. |
| C | 20/25 = 80% | 4 / 1 | 14/20 = 70% | Sufficient participation; agreement below threshold. |
| D | 0/25 = 0% | 20 / 5 | No evaluable ratings | Insufficient participation; agreement undefined. |
For this example, quorum requires at least 20 ratings. With 20 valid ratings, the agreement threshold requires at least 15 supporting votes. Retain the integer counts when applying the rule; display rounding should not determine whether an item passes.
Resolve eligibility and boundary cases before collecting ratings
Fix the eligible panel for each round and explain any item-specific expertise restrictions in advance. Excluding everyone who abstains from the eligible count would change the participation calculation; document the intended approach rather than choosing the denominator after seeing the result.
State whether a boundary such as 80% is inclusive, how whole-vote requirements are derived, and how withdrawals or replacements affect eligibility. Distinguish lack of expertise, a declared conflict, a skipped item, and a technical failure when those reasons matter to interpretation.
Specify the response when quorum or agreement is not achieved: a reminder, clarification, another round, discussion, or an unresolved classification. Keep that procedure consistent with the stopping rules. More votes can improve coverage, while substantive disagreement may require a different response.
If stakeholder groups have separate requirements, apply and report each group’s counts and decision. A pooled percentage cannot establish that every required group met its own participation or agreement rule.
Show the denominator behind every acceptance label
For each item and round, report the eligible count, substantive ratings, supporting ratings, explicit non-ratings, missing answers, both percentages, both thresholds, and final status. Preserve rating distributions and explanations when they affect interpretation.
Original reporting example
“Item B received 20 substantive ratings from 25 eligible panelists (80% participation); three abstained and two provided no item response. Fifteen of the 20 ratings supported the statement (75%). The item met both prespecified thresholds.”
Use a separate label for insufficient participation so readers can distinguish limited coverage from sufficient participation with inadequate agreement. When there are no valid ratings, report the agreement percentage as undefined rather than assigning zero support.
In Surveylet, configure the response labels and rating scale for the approved protocol, then reconcile the item counts in the relevant output. If a separate quorum calculation is needed, define it in the analysis plan and reporting specification and check the result against a worked example before launch.
In Real-Time Delphi, specify the time or snapshot at which each result is classified. A minimum-answer setting that controls when analytics become visible has a different purpose from a prespecified quorum rule for accepting an item.
Sources and scope
International Society for Heart and Lung Transplantation. Delphi Methodology Guidance for Authors of ISHLT Consensus Documents. Undated; accessed October 23, 2026.
ACCORD reporting guideline for consensus methods in biomedicine. PLOS Medicine. 2024;21:e1004326.
The calculations and reporting examples are Calibrum’s original illustrations. They do not establish universal Delphi thresholds and do not replace the study’s approved methodology.
Make every acceptance decision interpretable
Plan the participation counts, agreement rules, and reporting fields your team needs before invitations go out.