How to Review a Delphi Framework with Practitioners
Once a panel has agreed on a framework, ask the people who will use it to work through a realistic task. Their experience can reveal where the framework still needs clarification or testing.
An IT governance study added a separate appraisal
A November 2026 study of automation governance in UAE telecommunications combined standards and literature review, expert content review, and modified Delphi. Thirteen experts contributed to the Delphi, and nine practitioners from three operators appraised the resulting framework.
The early access report describes favorable practitioner judgments. It does not report evidence that deploying the framework improved operations. Those two questions call for different designs.
Give reviewers a job to do
Decide what the people using the framework need to be able to do: find the applicable rule, choose the next action, identify who owns it, or record enough evidence for the next person. Recruit for the roles, sites, and conditions the framework is supposed to cover.
Give everyone the same version and instructions. Use a routine case and an exception with a meaningful decision, such as a handoff outside normal hours. Ask participants to show how they would handle each case and where they had to make an assumption. Capture their first answers before group discussion if independent interpretation matters.
Define the questions and measures before the review. A usefulness rating can describe how reviewers felt about the framework. An observed task can show whether reviewers found the right instruction and recorded the required information. Report an unattempted task separately from an unsuccessful attempt.
Follow one responsibility through three stages
Imagine a set of agreed procedures for lending equipment at training centers. The panel endorses the requirement to name someone responsible for each return. The project team then tests how closing staff would use the framework after the front desk shuts.
| Review | Question to answer | Evidence to retain |
|---|---|---|
| Expert panel | Is a named person responsible for each return? | The item version, ratings, valid count, and acceptance decision. |
| Practitioner walkthrough | Can the closing team record a return when the front desk is closed? | The scenario, framework version, attempted steps, and any ambiguity the team found. |
| Field pilot | Do actual returns have a recorded owner by the next shift? | Observed returns, missing owner records, time period, and operating conditions. |
The walkthrough might show that two staff members interpret the after-hours role differently. Ask the responsible team to clarify it, log the change, and assess it again with the people who perform that task. If the revision changes the panel-approved responsibility, send the changed item through the planned approval process.
Show what was assessed and what happens next
Keep the original panel result and the practitioner observations side by side. For each issue, record the framework version, scenario and role, number of participants who attempted the task, what happened, and the proposed change. Explain how reviewers were selected, including any overlap with the Delphi panel.
Use the field pilot to answer a question the appraisal could not. In the equipment example, count actual returns with an owner recorded by the next shift and report the number of returns observed. If the question is whether the framework improves operations, plan an evaluation that can distinguish its effect from changes in staffing, workload, or recording practices.
Surveylet can collect expert ratings in one stage and structured practitioner responses in another. Match exports using the framework item and scenario IDs, and define how the team will log observations outside the survey. Keep review outcomes, revisions, and release decisions traceable in the report.
Source and scope
Shahtout A, Abu Bakar NA, Ab. Rahim NZ. Content validation and practitioner appraisal of a robotic process automation governance framework. Information. Early access, November 11, 2026. DOI: 10.3390/info17111136.
The equipment loan scenario and suggested review record are Calibrum’s original illustrations.
Plan the review before rolling out the framework
Decide who will assess the content, who will try using it, and what evidence is needed to move to a pilot.