RAND/UCLA Appropriateness Method
The RAND/UCLA Appropriateness Method (RAM) rates the appropriateness of clinical scenarios: typically whether a procedure, test or treatment is appropriate, uncertain or inappropriate for a defined patient presentation. Panelists rate on a 1–9 scale. The published method includes a discussion of the first round's results before a second rating. Delphi Studio does not run that discussion. Classification in this application follows the panel median and a disagreement test, not a percentage threshold. Whether a study that omits the discussion may still use the RAND/UCLA name is a methodological choice this page does not settle.
Setting it up in Delphi Studio
Two choices in Setup make a study RAND/UCLA:
- On Method, choose RAND/UCLA Appropriateness. This records the framework only. It does not choose the consensus method.
- On Rating & consensus, choose RAND/UCLA Appropriateness: median and disagreement. This is what makes the results use the median rule. Until it is chosen, statements are classified by the percentage rules of modified e-Delphi, and the Method screen points you to Rating & consensus.
The RAND/UCLA consensus method needs a 1–9 scale with an uncertain zone. Setup will not confirm the rules on any other scale. If the zones differ from the manual's 1–3 and 7–9, Setup warns that the Methods text should say so. The percentage fields are kept while RAND/UCLA is on, so switching back to the Modified Delphi method loses nothing. After launch, the scale, the zones, the consensus method, and the spread rule stay fixed. For RAND/UCLA, a recorded amendment of the consensus numbers is refused before that change is read. The disagreement test and the IPRAS constants are not part of that amendment. A direct change to them is refused because the study has launched. Stopping advice can still be amended with a reason. That change does not reclassify a round already rated. The full boundary is Launch and amendments. See Study design and Round design.
When to use it
| Appropriate | Less appropriate |
|---|---|
| Procedure indication panels | Competency agreement (use modified e-Delphi) |
| Diagnostic test ordering | Policy priority ranking |
| Treatment pathway review | Curriculum learning outcomes |
RAM items are usually scenario-based: a clinical vignette plus a proposed action to rate.
Example item:
68-year-old woman with stable angina, no diabetes, normal renal function. Rate the appropriateness of immediate coronary angiography within 24 hours.
The 1–9 scale
| Score | Meaning |
|---|---|
| 1 | Extremely inappropriate |
| 2 | |
| 3 | |
| 4 | |
| 5 | Equivocal |
| 6 | |
| 7 | |
| 8 | |
| 9 | Extremely appropriate |
The 4–6 band on the diagram is a zone on the scale. It is not, by itself, the result class Uncertain. With the default zones, appropriate starts at 7, so a median of exactly 6.5 is Uncertain in this product. The manual's most common approach calls 6.5 appropriate. A median of 3.5 is uncertain either way. Whether this product should change is open. That sentence is about the default zones, not about a study that has moved the zone edges. See Consensus analytics.
Panelists see the anchors the study sets for each zone. The words Appropriate and Inappropriate as results appear only on staff screens and in reports. The figures a panelist sees do not include that class.
How Delphi Studio classifies a scenario
| Result | Rule |
|---|---|
| Appropriate | Median 7–9, and the panel does not disagree |
| Inappropriate | Median 1–3, and the panel does not disagree |
| Uncertain | Median between the zones (above 3 and below 7), or a decided median with disagreement |
| Too few responses | Fewer numeric ratings than the study's minimum (7 by default) |
"Unable to rate" answers are left out of the count and still shown. Percentiles and the median use linear interpolation between ordered ratings (the inclusive, type 7 method). The manual never names that convention. Its printed examples match it. Another convention can change the class.
The table and the diagram describe the default zones. The median 6.5 qualification above applies to those zones.
The disagreement test
The study chooses one of two tests on Rating & consensus.
Interpercentile range adjusted for symmetry (IPRAS), the default and the manual's current method:
IPR = P70 − P30
AI = | 5 − (P30 + P70) / 2 |
IPRAS = IPRr + AI × CFA (IPRr 2.35, CFA 1.5)
The panel disagrees when the IPR is larger than the IPRAS. A value equal to the limit is not disagreement. A spread that sits off-centre (towards one end of the scale) is allowed to be wider before it counts as disagreement. IPRr and CFA can be changed before launch.
Tertile test. The panel disagrees when at least a third of the ratings fall in 1–3 and at least a third fall in 7–9. A count that is exactly one third counts. That matches the classic rule for a panel of 9. It is stricter than RAND Table 4 at panel sizes 10, 13, and 16. IPRAS stays the default. Whether to use Table 4 instead is open. This page does not adopt Table 4.
The results table shows the percent in each extreme zone, the median, the interquartile range, and how many people answered. It does not show the interpercentile range or the tertile counts. The consensus CSV has that same family of columns and no interpercentile-range column. See Consensus analytics.
Worked example
These nine ratings are synthetic. They are not a panel's results: 1, 2, 3, 7, 7, 8, 8, 9, 9.
| Measure | Value |
|---|---|
| Median | 7 |
| P30, P70 | 4.6, 8 |
| IPR | 3.4 |
| AI | 1.3 |
| IPRAS limit | 2.35 + 1.3 × 1.5 = 4.3 |
| Ratings 1–3, 7–9 | 3 low, 6 high |
Under IPRAS, the IPR (3.4) is within the limit (4.3), so there is no disagreement and the scenario is Appropriate. Under the tertile test, a third of the ratings (3 of 9) are in 1–3 and more than a third are in 7–9, so the panel disagrees and the scenario is Uncertain. The two tests can give different answers, so pre-specify one in the protocol.
What does not apply under RAND/UCLA
- The percentage Retain and Exclude thresholds, the IQR limit and the median condition are not used. Only the minimum number of responses carries over.
- The polarized rule is off: disagreement is judged by the test above.
- The optional cross-stakeholder condition belongs to the percentage rules and is not applied.
- The disagreement index stored for a 1–9 scale with at least three ratings is (P70 − P30) / 2.35. It is not the RAND/UCLA disagreement test. It is not a column on the results table and not a column in the consensus CSV. Under RAND/UCLA it does not decide the class.
Panel requirements
RAND/UCLA manuals traditionally recommend panels of about 7 to 15 members. That 7 to 15 is a panel size. The 7 in Delphi Studio is a different number: the minimum numeric ratings required before one statement is classified. Some studies run separate panels, for example an internal-medicine panel and a surgery panel, on the same scenarios. That pair is an illustration, not a citation of a particular manual passage.
- Delphi Studio classifies the whole panel together. Size & mix groups can describe the specialties, but a separate result per group or subgroup is not computed yet.
- A study that needs separate panels can run each as its own study, or report the limitation in its methods.
- The minimum number of responses (7 by default) is set on Rating & consensus; below it a scenario is "Too few responses".
Differences from the modified e-Delphi rules
| Aspect | Modified e-Delphi | RAND/UCLA |
|---|---|---|
| Scale | Any offered scale (1–9 by default) | 1–9 only |
| Decides the result | Percentage of ratings in a zone, plus a spread limit, on Modified Delphi: percentage and spread | The panel median, plus a disagreement test |
| Results named | Retain, Exclude, Uncertain, Polarized | Appropriate, Inappropriate, Uncertain |
| Disagreement | IQR or disagreement index limit; polarized pattern | IPRAS or tertile test |
| Typical items | Statements | Clinical scenarios |
Panelist suggestions are a per-round switch, off by default, under either method. The two classifications are not interchangeable. A percentage rule does not produce an appropriateness class, and the RAND/UCLA median rule does not produce Retain or Exclude as the labels a reader sees.
Shared behaviour
- Results are computed only after collection closes, from submitted answers, and stored as sealed results versions.
- Response rates below the study's thresholds (70% and 50% by default) add a note to results and the consensus report. Nothing is blocked.
- Kendall's W is computed, with a tie correction, and no rule uses it. See Consensus analytics.
- The stopping recommendation is advice only. Stability between rounds is not computed yet.
Reporting
RAM studies should report:
- Panel specialty mix and selection process
- Scenario development (literature review, focus groups)
- Per-scenario median, the disagreement test used and its result
- Round-over-round movement for contested scenarios
The consensus report and consensus CSV give each scenario's result under the study's own labels, with its permanent ID. There are no appropriateness-specific tables beyond these. See Reporting.
Next steps
- Comparing Delphi types: choose the right method
- Interpreting results: read results and decisions