Why a RAND/UCLA Item Can Remain Uncertain Without Disagreement
A panel can cluster around an uncertain judgment. The absence of disagreement describes the rating pattern; the median and the study’s decision rule determine the substantive classification. Report both so the result can be interpreted.
A recent panel found no final disagreement but retained uncertain items
Sussman and colleagues’ October 16, 2026 sepsis-recovery study used a modified RAND/UCLA process with 15 voting panelists. They assessed 18 practices for validity, importance, and resource intensity. Final ratings showed no disagreement, while ten practices were classified as valid and eight as uncertain.
The study’s validity construct concerned confidence that a practice is effective. Its findings illustrate a useful reporting distinction: the panel’s disagreement assessment and its substantive judgment remain separate outputs.
The rating patterns below are original Calibrum teaching examples. They are not the panel’s study data or recommendations about patient care.
Read the median alongside the disagreement result
The conventional RAM appropriateness classification combines a nine-point panel median with a defined disagreement assessment. A median in the high region with no disagreement supports an appropriate classification; a middle-region median remains uncertain. Disagreement also produces an uncertain classification under that framework.
The RAND manual describes the underlying approach. Record the version and any modifications used by the study, including fractional-median boundaries, percentile conventions, minimum valid ratings, and non-rating exclusions.
An uncertain classification can therefore arise through different routes. Panelists may cluster near the middle, or opposing ratings may satisfy the chosen disagreement rule. The classification alone cannot tell readers which pattern occurred.
Keep the rating construct explicit. A modified study using a validity scale should define that scale and its classification labels. Importance, appropriateness, confidence in effectiveness, and resource intensity should not become interchangeable because they share a numerical scale.
Compare four original nine-person rating patterns
These examples use nine valid appropriateness ratings and the standard IPRAS comparison: disagreement when the 30th-to-70th interpercentile range exceeds its symmetry-adjusted threshold. Percentiles use linear interpolation. The examples use integer medians, so they do not require a fractional-median boundary rule.
| Rating pattern | Median | Disagreement under IPRAS | Classification |
|---|---|---|---|
| All nine panelists rate 8. | 8 | No | Appropriate |
| All nine panelists rate 5. | 5 | No | Uncertain |
| Four rate 1, one rates 8, and four rate 9. | 8 | Yes | Uncertain |
| All nine panelists rate 2. | 2 | No | Inappropriate |
In the polarized pattern, the percentile limits are 1 and 9. The interpercentile range is 8, its central point is 5, and the IPRAS threshold is 2.35. Because 8 exceeds 2.35, disagreement is present despite a high median.
In the all-5 pattern, dispersion is zero. The panel is highly consistent about a middle-scale judgment. That consistency does not provide a favorable appropriateness judgment. See the RAM setup guide for the component calculations and configuration checks.
Preserve validity, importance, and resource requirements separately
A practice may be considered important while confidence in its effectiveness remains uncertain. Another may receive a favorable validity judgment but require substantial staffing or follow-up work. Those profiles support different discussions.
Define the direction and meaning of every scale. A high importance score indicates a stronger positive importance judgment, while a high resource-intensity score indicates greater resource requirements. Combining the two into a single favorable-score total can obscure the trade-off.
If a prioritization stage considers only items meeting a validity requirement, state that eligibility rule and retain the excluded items in the full results. A plot containing only selected items should disclose the selection rather than appear to summarize the entire questionnaire.
Use comments and discussion records to explain the conditions behind judgments. Keep those explanations linked to the relevant item and dimension. Resource ratings also need a setting and assumptions about staff, infrastructure, and any downstream work. See multidimensional RAM reporting.
Give readers the reason for uncertainty
For each item, retain its identifier, wording version, rating construct, round, valid-rating count, median, distribution, disagreement definition, calculated result, and final classification. Include non-ratings and missingness separately. A numeric middle response remains a rating; an explicit inability to rate is a different response.
Original reporting examples
“The item was classified as uncertain because the median was 5. No disagreement was identified under the prespecified IPRAS comparison.”
“The item was classified as uncertain despite a median of 8 because the distribution met the prespecified disagreement criterion.”
Where ratings change after discussion, retain the before-and-after results and the discussion summary. Explain whether wording or participation changed. Avoid attributing every change to persuasion when the item or responding group also changed.
In Surveylet, review the RAM-related outputs and their coding against the approved protocol. General range-consensus status is a separate result and should not substitute for a confirmed RAM classification. Agree any derived classification or custom combined reporting table before analysis.
The aim is a record that makes uncertainty interpretable and preserves minority concerns, evidence limitations, and context alongside the quantitative result.
Sources and scope
Sussman JB, Hogan CK, Whittington T, et al. Best Practices for Optimizing Recovery from Sepsis; Results of an Expert Panel. Journal of General Internal Medicine. Published October 16, 2026.
Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. RAND; 2001.
The sepsis study is an example of a documented modification using separate rating dimensions. The nine-person patterns and reporting text here are original teaching illustrations, not observed clinical results or a universal rule for every Delphi design.
Plan a result readers can interpret
Define the scales, disagreement assessment, classification boundaries, and reporting fields needed for your RAND/UCLA study.