Skip to content
Quality measurement and expert judgment

Selecting Quality Measures with a Modified Delphi Study

A useful quality measure must address something that matters, be practical to collect, and represent the quality it claims to assess. Rate those judgments separately, explain how the shortlist was chosen, and test how the selected set works in practice.

A new study separates measure ratings from final prioritization

Becker and colleagues’ November 21, 2026 Mayo Clinic report describes an environmental scan followed by a three-round modified Delphi process. Participants rated importance, feasibility, and validity before final ranking and consensus review. The project prioritized 10 of 31 candidate measures for pharmacist-led inflammatory bowel disease care.

The abstract identifies shared documentation and electronic health record tools as next steps. This illustrates the transition from selecting measures to implementing them; it does not establish their effects on care.

The planning approach below is Calibrum’s own illustration. Its numbers and decision rule are fictional and are not the study’s methods or results.

Give each candidate a purpose and a traceable source

Define the service, population, and intended use before assembling a measure bank. Improvement within one service, comparison across organizations, and external accountability can require different evidence and specifications.

Give every candidate a stable ID, definition, source, and rationale. Record whether it was adopted, adapted, or newly proposed. Check for duplicates and for gaps in the aspects of quality the project intends to cover.

Include perspectives needed to judge both quality and data collection. Explain panel eligibility, roles, and representation. A convenient group of respondents may overlook operational constraints or the experiences of people receiving the service.

Ask three questions before calculating an overall result

Define each dimension in the questionnaire: importance asks whether measuring this aspect of quality matters; feasibility asks whether the specified data can be collected in the intended setting; validity asks whether the proposed measure represents the intended aspect of quality. Panel judgment about validity still needs empirical testing.

For this fictional administrative-service example, 20 panelists provide valid 1–9 ratings for every dimension. The illustrative eligibility rule requires at least 80% to score 7–9 on each dimension: 16 of 20. This is an example rule, not a universal cutoff or a recommendation for every study.

Original example: counts rating 7–9, with 20 valid ratings per dimension
Candidate measureImportanceFeasibilityValidityEligibility under the example rule
Request acknowledgments sent within the defined service window18/2017/2018/20Eligible for prioritization
Resolution time across all participating services18/2011/2017/20Address the feasibility concern
Positive feedback used as evidence of accurate case resolution19/2017/2012/20Review what the measure represents

Averaging the three dimensions would conceal the reason a candidate needs work. Retain distributions, comments, and valid response counts for each dimension. Predefine missing-answer handling, disagreement checks, revision rules, and any stakeholder requirements. See multidimensional Delphi criteria.

Select a useful set from the eligible measures

Eligibility and priority answer different questions. Several measures may satisfy the rating rules while the team has resources to pilot only a few. State who ranks the eligible measures, how many can be selected, and how ties will be handled. The priority-ballot guide offers a worked selection example.

Review the set for duplicated information, collection burden, and coverage. A shortlist dominated by easy documentation counts may leave important service outcomes unmeasured. If coverage requirements affect selection, establish them in advance and report their effect on the final order.

Retain distinct statuses for eligible, selected, revised, deferred, and excluded candidates. If a later committee changes the shortlist, record its authority and rationale. A deferred measure may remain important while its data problem is resolved.

Pilot data collection and report the path to the final set

For each selected measure, specify the numerator, denominator, exclusions, measurement window, data source, and responsible owner. The audit-standards guide explains that specification step. Record missing information explicitly and check that different sites interpret the definition consistently.

Use a field pilot to examine data availability, extraction effort, and agreement between reviewers. Investigate whether results reflect service quality or mainly differences in documentation. Predefine what findings would lead to revision, renewed panel review, or deferral.

Report candidate sources and versions, panel participation by round, feedback, each dimension’s result and denominator, revisions, ranking rules, and final dispositions. Distinguish agreement on the proposed measures from evidence about their performance after implementation.

For a Surveylet project, plan separate rating questions where needed and preserve the candidate ID in the questionnaire and exports. Test feedback and non-rating behavior before launch. Agree any combined decision table, priority analysis, and measure specification within the reporting scope.

Source and scope

Becker MD, Groehler PN, Bruining DH, et al. Developing and prioritizing standardized quality measures for pharmacist-led care of inflammatory bowel disease with a modified Delphi process. American Journal of Health-System Pharmacy. Published November 21, 2026; accepted manuscript, zxag409.

The research summary uses the publisher’s abstract. Detailed thresholds and panel counts are not inferred from it. The administrative-service example and planning suggestions are original illustrations. The cited report is a methods example, not evidence of Surveylet use.

Plan how your quality measures will be selected

Define the panel, separate rating dimensions, selection rules, and reporting outputs before the first round opens.