Where AI Can Help a Delphi Study—and What Researchers Must Still Review
AI can help organize comments, retrieve material, and draft clearer wording for a research team to review. The methodological question is whether that assistance preserves the evidence, meaning, and independence of the expert judgments being collected.
A comparative study offers an early test of assisted revision
Nello and colleagues’ July 13, 2026 study assigned 50 panelists to human-revision and AI-assisted groups rating 28 RNA-medicine statements. Researchers verified citations and approved or amended AI drafts. After round 2, 24 of 28 statements met consensus in the AI-assisted group, compared with 26 of 28 in the human group.
The groups differed at baseline, and the study did not prospectively measure workload, turnaround time, or cost. It supports further evaluation of reviewed drafting assistance. It does not establish superiority, equivalence, or a quantified efficiency saving.
The practical examples below are Calibrum’s original suggestions for research teams considering a similar task.
Define a specific task and a person responsible for reviewing it
Choose a bounded task that the team can inspect. For example, use an approved collection of anonymized comments to suggest possible ambiguities, or ask for candidate wording that addresses a documented concern. Keep the original comments and statement available to the reviewer.
Decide what the output is allowed to change. A request to improve readability should preserve the population, action, comparison, conditions, and strength of the recommendation. A request to reconsider evidence needs a different review process and the relevant subject expertise.
AI-generated wording and summaries are contributions to the team’s preparation work. The expert ratings remain the judgments submitted by the recruited panel. Replacing missing panelists with generated responses would change the study design and cannot be treated as ordinary completion of that panel.
Before sending comments or evidence to another system, establish whether the planned data use is covered by the project’s approvals and data-handling arrangements. Use the minimum material needed and check for identifiable details in free text.
Inspect how an apparently clearer sentence changes the decision
Consider an original recommendation: “Offer a follow-up discussion to adults who report difficulty managing their recovery.” A draft revision might add timing or make the action mandatory. These changes affect what panelists are being asked to endorse.
| Proposed change | Possible effect | Review needed |
|---|---|---|
| Add “within seven days.” | Introduces a deadline and possible resource implications. | Check the source and determine whether timing needs a separate rating. |
| Replace “offer” with “provide.” | Changes an offer into a requirement for completion. | Confirm the intended action, including how a patient’s choice is handled. |
| Remove “who report difficulty.” | Expands the eligible population. | Review scope, supporting evidence, and whether the recommendation still addresses the same question. |
| Correct a reference or clarify a defined term. | May improve traceability or comprehension. | Verify the source passage and retain a record of the correction. |
Ask the reviewer to explain the effect of each accepted edit. If the wording introduces a substantive condition, preserve the original version and decide whether the panel needs to assess the revised statement under the protocol.
A more agreeable sentence may simply be less specific. Compare the scope of the before-and-after versions before interpreting a higher consensus percentage as an improvement.
Retain the evidence and the human decision behind each revision
Use a revision record containing the item identifier, original wording, relevant comments, candidate edit, rationale, evidence consulted, reviewer decision, final approved wording, version, and date. Retain rejected suggestions when they explain a substantive decision.
Open each cited source and check that it supports the particular claim or wording change. Confirm the population, intervention, setting, and limitations relevant to the statement. A valid DOI proves that a publication exists; the link alone cannot establish support for a new recommendation.
Inspect summaries for omitted uncertainty and minority views. A comment mentioned by one panelist can identify an important exception. Frequency of mentions should not automatically decide whether it is retained in the feedback.
Record the tool and model or version, date of use, task instructions, source collection, and extent of human editing needed for reproducibility. Where prompts or outputs cannot be shared, explain the restriction and preserve a suitable internal record.
Evaluate quality, effort, and consensus as different outcomes
Before a pilot, decide what successful assistance would mean. Possible measures include unsupported claims, substantive meaning changes, overlooked concerns, reviewer effort, time per approved revision, and panelist comprehension. Define how errors will be identified and who will adjudicate them.
Measure preparation and review effort as well as drafting time. Work saved in producing a first draft can be offset by citation checking or corrections. Record the complete workflow before making an efficiency claim.
Report agreement rates with valid item counts and baseline results. Explain wording changes and changes in panel participation. If different groups or versions are compared, describe those differences and the limits they place on causal interpretation.
Disclose where AI contributed to the study, what it received, what it produced, and who reviewed the output. Apply the reporting requirements of the journal, institution, and study approvals. The Delphi reporting guide provides a broader record of item development, feedback, revisions, and decisions.
Plan the revision record alongside Surveylet’s expert ratings, item comments, and exports. Agree responsibility for evidence review, data use, and final wording before another round opens. Any AI integration or additional reporting workflow should be scoped separately for the project.
Sources and scope
Nello E, Tedone F, Caproni E, Cafiero D, Manellari S, Rocco P. A Controlled Comparison of Human and AI-Assisted Automated Revision of Delphi Statements on RNA-Based Medicines: Parallel, 2-Arm Study. JMIR Medical Informatics. 2026;14:e90228. Published July 13, 2026.
Consult the ACCORD reporting guideline for reporting the underlying consensus process. The review examples and evaluation suggestions on this page are original illustrations. The cited study evaluated a specific workflow and does not establish that an AI feature or integration is available in Surveylet.
Keep the research process reviewable
Agree the questionnaire, evidence-review responsibilities, revision history, and reporting outputs needed for your Delphi study.