When External Review Changes a Delphi Definition
External review can uncover an ambiguity after a panel has reached agreement. A clear process links the review finding to the revised wording and the assessment needed to support the final definition.
A new consensus paper shows why review can reopen a decision
A global multisociety consensus on acute-on-chronic liver failure, published October 30, 2026, describes working groups drawn from five liver societies. Following an open meeting and feedback from attendees and US FDA representatives, 12 statements were revised and assessed in a second Delphi panel.
The authors also reported evidence levels and recommendation strengths alongside the agreement process. This provides a useful methodological example of documenting external input and returning revised statements for assessment. Regulatory participation in a discussion should be described by its actual role; it does not establish regulatory approval.
The review workflow and fictional examples below are Calibrum’s original planning suggestions. They are intended for consensus-study design and reporting.
Classify what the reviewer’s suggestion changes
Ask whether a reasonable panelist could judge the revised wording differently. Changes to the population, action, cutoff, timing, exception, or measurement can alter the proposition being rated. Agree who makes this assessment and how the decision is recorded.
| Reviewer suggestion | What changes? | Record and next step |
|---|---|---|
| Correct a spelling error in an unchanged definition. | Presentation; the intended meaning remains the same. | Retain the earlier wording and record the editorial correction and reviewer approval. |
| Add a follow-up exercise to the definition of “completed onboarding.” | The conditions needed to meet the definition. | Create a new wording version and apply the protocol’s review and re-rating process. |
| Change the agreement rule after seeing which statements passed. | The study’s classification method. | Preserve the original rule and results; justify and disclose any amendment and additional analysis. |
Classify uncertain cases for explicit methodological review. The log should show the original wording, proposed revision, reason, source of the suggestion, decision maker, and whether further panel assessment is required. External reviewers can identify a problem while the study team remains responsible for the process used to resolve it.
Keep each result attached to the wording that earned it
Use a stable item identifier and a separate version label. In a fictional community-service study, D07-v1 defines completed onboarding as attendance at an introductory session. An external reviewer proposes requiring both attendance and a follow-up exercise. Save that proposal as D07-v2, linked to D07-v1.
The earlier vote describes support for D07-v1. Record a fresh result for D07-v2 if it is assessed, with its own valid-rating count, supporting votes, distribution, and classification. A visual comparison or a percentage copied from the first version cannot establish agreement on the new requirement.
Retain the review materials and the exact text shown during re-rating. If only selected statements return to the panel, explain the selection rule and account for the disposition of every reviewed item. Record statements retained unchanged, revised, withdrawn, and unresolved.
Where review produces a new statement after the planned final round, apply the protocol’s extension or unresolved-item procedure. Document any amendment before reporting the final list. See sustained consensus across rounds when a study requires repeated agreement on comparable wording.
Give the revised proposition a defined assessment
Explain the review finding and show the revised wording with enough context for an independent judgment. Predefine whether panelists see the prior version, its results, the review rationale, and anonymized comments. Preserve the feedback package so the later report can explain what participants considered.
State the rating scale, supporting categories, exact threshold operator, denominator, participation requirement, handling of non-ratings, and closing rule. A rule of at least 80% support has a different boundary from a rule of more than 80%. These illustrative values are design choices; the appropriate rule needs a study-specific rationale.
Report who was invited to the renewed assessment and who responded. Compare the panel’s composition with the earlier stage, including relevant stakeholder groups and any replacements. A second assessment should have a transparent participation record even when the study originally intended to retain the same panel.
Keep methodological amendments distinct from revised statements. If an analysis rule changes, preserve the result under the original rule and identify the amended or exploratory analysis. Readers should be able to follow both the change in wording and any change in classification.
Report evidence, agreement, and external input in their own terms
A useful final record connects the evidence summary, statement version, panel result, recommendation strength where assessed, external-review comment, and resulting decision. Explain how each field was produced. Agreement percentages describe judgments from the participating panel; evidence grades describe an appraisal under a stated framework.
Identify the external reviewers’ selection, interests, remit, and influence on revisions. Attribute editorial decisions to the editors or steering group and voting results to the panel. Use institutional names only for the involvement or endorsement actually documented.
Surveylet supports structured ratings, item-linked comments, rounds, and exports. Plan how the team will maintain the review log and version crosswalk alongside those outputs. Agree any tailored reporting deliverables with Calibrum before the study starts.
The Delphi reporting guide can help specify the wider record, including participant flow, feedback, unresolved items, and protocol changes.
Source and scope
EASL, AASLD, APASL, ALEH, and SOLDA. Acute-on-chronic liver failure: a global multisociety consensus on definitions. Hepatology International. Published October 30, 2026.
The paper informs the introductory research example. The change classifications, onboarding example, and proposed review record are original Calibrum guidance. The citation is a methodological example and does not identify a Surveylet study.
Plan how review will affect the final record
Define reviewer roles, statement versions, renewed assessment, and reporting outputs before external comments arrive.