Does Consensus on a Checklist Validate the Workflow?
A panel may agree on every action in a checklist while the order of those actions, the exceptions, and the handoffs still require review. Describe the scope of agreement before presenting the checklist as a complete workflow.
A recent study makes the boundary explicit
Ryan and colleagues’ October 26, 2026 robotic-surgery Delphi study produced 37 consensus statements across four rounds. The researchers then arranged the statements into an operational algorithm. They explicitly distinguish agreement on the individual statements from the sequence and branching, which were not submitted to the panel.
The authors identify a need for further sequencing work and simulation and clinical validation. The methodological lesson is useful beyond surgery: agreement on the components and assessment of their arrangement are separate research tasks.
The examples below are Calibrum’s original, fictional illustrations of that distinction. They describe planning and reporting decisions rather than instructions for a clinical procedure.
Specify the object of each judgment
Before writing questions, decide whether the panel is assessing the importance of an action, its appropriateness under stated conditions, its position in a sequence, or responsibility for carrying it out. A positive rating on one question cannot answer all the others.
For each proposed workflow, identify the actions, decision points, dependencies, parallel activities, exception routes, and handoffs that need assessment. Link each judgment to the wording and conditions shown to the panel. Several individually reasonable actions may become difficult to deliver when combined.
Ask whether every affected professional or stakeholder group is represented. When a process assigns work to a group, record that group’s input and any further endorsement needed. Overall agreement should be interpreted alongside the composition of the panel.
Make an agreed action and an inferred connection distinguishable
Imagine a fictional Delphi about reviewing supplier requests. The panel has agreed that requests should be checked for completeness and that an authorized reviewer should approve a decision. The team now proposes a process map.
| Component | What was assessed? | Record or next question |
|---|---|---|
| R04: Check request completeness. | The panel rated the stated action. | Retain the item version, round, denominator, and classification. |
| T01: Return incomplete requests before reviewer assignment. | The team proposed a dependency between actions. | Determine whether the dependency needs panel assessment and exception handling. |
| A02: Assign completeness checking to the receiving team. | A local manager proposed a responsibility. | Confirm authority, capacity, and input from the people doing the work. |
| T02: Run two reviews in parallel for urgent requests. | The team proposed a conditional route. | Review eligibility, staffing, conflicting decisions, and the return to the main process. |
The map should preserve those distinctions even when its visual presentation uses identical boxes and arrows. A diagram can otherwise imply that every connection received the same review as the underlying statements.
Keep the original statement alongside any operational wording. If a new timing requirement, population, or condition changes its meaning, record that change and determine the appropriate review under the protocol.
Review the sequence and test how people use it
If the study intends to claim agreement on a workflow, plan questions that assess its sequence and key decisions. Review manageable sections with enough context to make dependencies understandable. Give panelists a way to explain missing steps, ambiguous conditions, and alternative routes.
Original review prompts
“Can the next action begin before this check is complete?”
“Who can make the decision when the usual reviewer is unavailable?”
“Which exception would send this request back to an earlier stage?”
Define whether an accepted change requires another review of connected actions. Moving one step may affect information availability, accountability, or timing elsewhere. Retain the version of the complete process shown in each assessment.
Then plan a suitable application test. Fictional cases, walkthroughs, or simulations can examine interpretation, missed actions, completion time, and handoffs. Define the setting and intended outcomes before testing. State separately what the panel endorsed and what the application test established.
Testing a workflow can lead to revisions. Preserve the original result, explain the change, and record whether another panel assessment or operational approval is required. See vignette-based application testing for a related approach.
Publish the chain from panel item to process version
A useful record links each process component to its source item, wording version, panel judgment, later adaptation, and approval decision. Identify elements added by the research team, implementation team, or local service.
Report the scope of the original rating, required stakeholder groups, eligibility and denominators, treatment of non-ratings, and the rule used to classify the result. For sequence review, also retain the process version, branches assessed, unresolved exceptions, and any decisions made after the formal ratings.
Keep application-test findings and subsequent implementation outcomes in their own reporting fields. An agreed checklist, a usable process, and improved outcomes require different evidence. Describe the evidence actually available for each.
Surveylet can collect structured ratings and item-linked comments and provide exports for analysis. Agree any process mapping, sequencing exercise, approval log, or validation work as part of the project’s methods and reporting scope. The implementation-planning guide covers actions, ownership, and evaluation after the Delphi.
Sources and scope
Ryan G, Shakir T, Francis N. Emergency undocking in robotic surgery: an international Delphi consensus study. Journal of Robotic Surgery. 2026;20:1135. Published October 26, 2026.
Consult the ACCORD reporting guideline for the underlying consensus process. Calibrum’s supplier-review scenario, prompts, and record examples are original planning illustrations and should be adapted to the study’s approved methods.
Define the scope of agreement before the first round
Plan the items, sequence questions, review stages, and reporting outputs needed for your proposed workflow.