From RAND/UCLA Ratings to a Decision Tree
A decision tree can make a large set of appropriateness judgments easier to read. Keep the original ratings and classifications close enough for readers to check where the tree changes the result.
A panel rated 256 scenarios before the authors summarized them
In a RAND/UCLA study of pediatric stroke scenarios, nine panelists rated 128 hypothetical scenarios in each of two age groups over two rounds. The authors used classification trees to summarize the panel’s judgments and retained fuller scenario tables in the supplement.
The tree describes patterns in the panel’s classifications. Performance against those classifications answers a different question from how the tree performs in new cases. The example below uses fictional service requests to demonstrate the distinction.
Estimate how many judgments the panel must make
Write the action being rated, then list the conditions that can reasonably change its appropriateness. Record each condition’s possible values and an operational definition. Give each valid combination a stable ID and version, and explain any excluded combinations.
Multiply the number of values per condition and account for each action rated. A fictional expedited-review study varies urgency, documentation, and reviewer coverage, with two values each. That gives eight scenarios. Nine panelists rating all eight in two rounds would supply 144 ratings for one action if they completed every item.
Pilot a representative block before launch. Time the rating task and check where context is missing or judgments become repetitive. If the full matrix is impractical, narrow the question or plan assignments so each case can still meet its participation and analysis rules. Report the valid count for every case.
Classify each scenario before simplifying
Retain the scale, rating distribution, median, disagreement rule, and final classification for every scenario and round. Record non-ratings and missing answers separately. The Surveylet RAM setup guide explains how to check the selected calculation before using it in a study.
The fictional example uses the classic nine-person disagreement rule: at least three ratings in 1–3 and at least three in 7–9. With all nine valid scores, disagreement makes the result uncertain regardless of the median. Otherwise a median of 1–3, 4–6, or 7–9 gives inappropriate, uncertain, or appropriate, respectively. The worked file uses this rule for exactly nine valid ratings.
Show where the short rule and the panel disagree
For the fictional action, the short rule labels standard requests inappropriate. It labels time-sensitive requests appropriate when their documentation is complete and uncertain otherwise. This proposed summary omits reviewer coverage.
| Scenario | Median | Ratings in 1–3 / 7–9 | Panel result | Short rule |
|---|---|---|---|---|
| S05: Time-sensitive, incomplete, limited coverage | 5 | 2 / 1 | Uncertain | Uncertain |
| S07: Time-sensitive, complete, limited coverage | 8 | 3 / 6 | Uncertain (disagreement) | Appropriate |
| S08: Time-sensitive, complete, available coverage | 8 | 0 / 8 | Appropriate | Appropriate |
The rule matches seven of eight scenario classifications (87.5%). It changes S07 from uncertain to appropriate even though three panelists rated that scenario in the lowest band. Show the exception alongside the rule. The team can add a branch, refer readers to the case table for this combination, or decide the short rule is too coarse for its purpose.
Keep a count of mismatches and inspect which kinds of errors matter. Describe how a fitted tree was built and checked, and distinguish its fit to the scenarios used to build it from results on held-out or new scenarios. A tree that reproduces a panel’s judgments still needs separate assessment for its intended use.
Download a scenario matrix and results table
The workbook includes eight fictional cases, nine editable scores per case, and a results sheet that compares the panel classification with the short rule. Try changing a score or leaving one blank to see how the median, disagreement result, and comparison respond. The example flags a case when it lacks nine valid scores.
Download the RAND/UCLA scenario matrix workbook (Excel)
For your own Surveylet project, assign scenario IDs before building the questionnaire and keep the same IDs in the exports. Record the exact disagreement rule and check the outputs against known examples. Agree any additional model fitting, comparison, or validation as part of the analysis plan.
Sources and scope
Carlos-Gil AM, Molina-Linde JM, Sánchez-Álvarez MJ, et al. Thrombolysis and thrombectomy in acute ischemic stroke: A RAND/UCLA consensus. Anales de Pediatría. November 2026 issue, article 504397. DOI: 10.1016/j.anpedi.2026.504397.
Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. RAND Corporation, 2001, chapter 8. The workbook and request scenarios are original teaching examples.
Plan the cases and reporting together
Define the scenarios, rating workload, and results you need to preserve before simplifying the findings.