Multi-round worked example
This example walks through three synthetic statements, showing how ratings become measures, classes, dispositions, and feedback. The ratings were written for this page. They are not a panel's results.
A figure, such as a median, is a statistical result. The badge, such as Uncertain, is the item classification. Advance, Revise, and Hold are researcher dispositions. Rated again is a plan tile. What confirmation does is the next-round action. Those are five different things. The decision reference is Interpreting results.
Setup
| Parameter | Value |
|---|---|
| Panel | 10 experts (all respond in both rounds) |
| Scale | Likert 1–5: disagree 1–2, uncertain 3, agree 4–5 |
| Retain | ≥ 70% rating 4–5, with IQR ≤ 1 |
| Exclude | ≥ 70% rating 1–2, with IQR ≤ 1 |
| Minimum responses | 7 |
| Polarized | ≥ 30% in each of 1–2 and 4–5, ≤ 20% rating 3, from 6 ratings |
| Rounds | 3 planned |
These figures are this study's rule, and they are the seeds for Likert 1–5. They are not a standard this page recommends, and this page does not say whether a study should adopt them. Choosing Likert 1–5 sets the zones to 1–2, 3, and 4–5, and it sets the spread limit to an interquartile range of 1. The median condition starts off. Polarized starts at 30 percent in each extreme, at most 20 percent in the middle, from at least 6 ratings. The built-in scale is 1–9. This study chose Likert 1–5.
Delphi Studio computes the median and quartiles by linear interpolation between ordered ratings (R type 7, Excel's QUARTILE.INC). With 10 ratings sorted as x1 to x10, the median is the mean of x5 and x6, Q1 = x3 + 0.25 × (x4 − x3), and Q3 = x7 + 0.75 × (x8 − x7). The tables below keep those calculated values. On the panelist block, a non-integer is shown to one decimal. Percentages count numeric ratings only. Unable to rate is shown but left out of the denominator.
Statement A — retained in Round 1
S01: "Annual competency review should be mandatory for telehealth nurses."
Round 1 ratings for statement A
| Rating | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Count | 0 | 0 | 1 | 3 | 6 |
Sorted: 3, 4, 4, 4, 5, 5, 5, 5, 5, 5.
| Measure | Value |
|---|---|
| Median | 5 |
| Q1 / Q3 | 4 / 5 |
| IQR | 1 |
| Rated 4–5 | 90% (9 of 10) |
| Rated 1–2 | 0% |
| Class | Retain |
Both retain conditions are met: 90% is at least 70%, and an interquartile range of 1 is within the limit of 1. The plan tile is Retained. The disposition is Hold. S01 is settled and is not rated again.
Statement B — uncertain in Round 1
S02, Round 1 wording: "Nurses should read ECGs during telehealth visits."
Round 1 ratings for statement B
| Rating | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Count | 1 | 2 | 4 | 2 | 1 |
Sorted: 1, 2, 2, 3, 3, 3, 3, 4, 4, 5.
| Measure | Value |
|---|---|
| Median | 3 |
| Q1 / Q3 | 2.25 / 3.75 |
| IQR | 1.5 |
| Rated 4–5 | 30% (3 of 10) |
| Rated 1–2 | 30% (3 of 10) |
| Rated 3 | 40% |
| Class | Uncertain |
Neither rule is met, and the interquartile range of 1.5 is over the limit of 1. S02 is not Polarized. Both extremes reach 30%, but 40% of ratings sit in the middle, above the 20% limit. While another round exists, the plan tile is Rated again and the disposition is Advance. That is not Revise.
Uncertain leads to Rated again and Advance. A reword is not on this path.
A separate Item bank revision
Advance would rate the same wording again. The sentence below is a different action. This illustration saves a clearer wording in the Item bank and cites S02's stored Round 1 result. That citation is not the classification, and confirmation does not carry it. Confirmation assigns only Rated again rows, and it does not write new wording.
S02 v2: "Nurses should recognize ECG red flags during telehealth visits and escalate per protocol."
Round 2 ratings of the new wording
| Rating | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Count | 0 | 0 | 2 | 5 | 3 |
Sorted: 3, 3, 4, 4, 4, 4, 4, 5, 5, 5.
| Measure | Value |
|---|---|
| Median | 4 |
| Q1 / Q3 | 4 / 4.75 |
| IQR | 0.75 |
| Rated 4–5 | 80% (8 of 10) |
| Class | Retain |
Retain is the class of these new ratings. The plan tile for that result is Retained, and the disposition is Hold. It is not the outcome of the Round 1 Uncertain class.
Statement C — polarized, rated again
S03: "Patients may receive prescriptions without synchronous video if audio quality is sufficient."
Round 1 ratings for statement C
| Rating | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Count | 3 | 1 | 0 | 1 | 5 |
Sorted: 1, 1, 1, 2, 4, 5, 5, 5, 5, 5.
| Measure | Value |
|---|---|
| Median | 4.5 |
| Q1 / Q3 | 1.25 / 5 |
| IQR | 3.75 |
| Rated 4–5 | 60% (6 of 10) |
| Rated 1–2 | 40% (4 of 10) |
| Rated 3 | 0% |
| Class | Polarized |
S03 is neither retained (60% is below 70%, and the interquartile range is far over 1) nor excluded (40% is below 70%). It is then checked for polarization: 60% and 40% in the two extremes are each at least 30%, nobody rated 3, and there are 10 ratings. So its class is Polarized rather than Uncertain. A median of 4.5 on its own would have hidden two opposed camps.
The plan tile is Rated again and the disposition is Advance. Polarized is not a clarity queue. Panelists do not see the split as a histogram. This statement is not reworded.
Feedback in the next round
Where the wording is unchanged, as with S03, feedback can show the previous round's current results: the median, Middle half of ratings, the zone shares including In between, the person's own earlier rating, and comments the team chose. It does not draw a histogram, it has no row labelled IQR, and it does not show the classification. The introduction still defines Status. There is no Status row. Those figures are the current results, not a snapshot from when the round opened.
For S02 v2, the panelist reads that the statement was reworded, so the earlier numbers are not shown. There is no Round 1 median, middle half, zone share, earlier rating, or comment.
After Round 2
| Statement | Result so far | Next |
|---|---|---|
| S01 | Retain (Round 1) | Hold. Tile Retained. |
| S02 v2 | Retain on the new wording | Hold. Tile Retained. Not the Round 1 outcome. |
| S03 | Polarized (Round 1), not settled | Round 3, already in the plan |
Two of three statements are settled: S01, and the separate revision S02 v2. S03 is not settled. The stopping display floors 2 of 3 to 66 percent. The 90 percent seed does not recommend stopping. That advice does not send S03 on. Round 3 is already in the plan of three rounds. S03 continues because a round remains and the statement is unsettled. Round 3 is the last planned round. If S03 is neither retained nor excluded there, and it did not come from a clarity review, the Decision-stage label is Final: no consensus.
Kendall's W is reported for the analyst. It measures how similarly panelists order the statements in one round. No classification rule uses it. This page does not give a value for these statements.
Takeaways
| Concept | Shown by |
|---|---|
| Share and spread | S01 is Retain, tile Retained, disposition Hold |
| Unsettled while a round remains | S02 is Uncertain, tile Rated again, disposition Advance |
| A reword is separate | S02 v2 cites a stored result. Its feedback shows no Round 1 figures |
| Polarized is not a reword | S03 is Polarized, tile Rated again, disposition Advance. No histogram |