Consensus analytics
Delphi Studio computes a fixed set of measures for every statement in a rating round and classifies each statement against the rules the study set before launch. The same rules drive the live preview on Rating & consensus and the results of a real round, so what the team tried before launch is what the round applies. Investigators still choose the rules and interpret the outcome.
Consensus and agreement
Consensus, on this page, is the class the study's rule gives one statement. Agreement is the share of numeric ratings that fell in the agree zone. A high agreement percentage is not itself the class.
Concordance is a third idea. Kendall's W, lower on this page, measures how similarly people ordered the statements inside one round. It is not an item-level consensus rule, and it is not stability. Stability can be configured. Nothing computes it.
Appropriateness is the RAND/UCLA class, median first, and only when the study is using that method. It is not the percentage rule.
From ratings to a decision
Results are never computed while a round is collecting. Once collection closes, someone who can edit Results chooses Compute results. That edit permission belongs to the principal investigator, the steering committee role, and the biostatistics role. A coordinator cannot compute results. A steering member whose only role is to confirm can read results and cannot compute them. When the action finishes, a results version is listed, or the screen says why it was not.
Advance, Revise, and Hold are decision words for a round that has another round after it. They are not the classification statuses below. The last round uses a separate set of final-result labels. Read Interpreting results before mapping a class to a decision.
- Only submitted responses count. A statement someone skipped has no answer.
- The Analysis plan's partial-responses rule decides whether a submission that answered too few statements is counted. Results always say how many people submitted, how many were counted and how many were left out.
- Each computation is saved as a results version. A version is sealed when it is written and checked every time it is read. If the check fails, the page shows no figures, and no decision or file can be made from it.
- Computing again asks why, and writes a new version that replaces the old one as the current result. Earlier versions stay listed.
Who confirms what the rules decided, and who decides a clarity or steering review, are in From a result to the next step. The sequence after the results is Between rounds.
Measures per statement
| Measure | What it is |
|---|---|
| n | Numeric ratings counted |
| Unable to rate | Answers of "Unable to rate", shown beside the figures |
| Agree, disagree and middle shares | The share of numeric ratings in the agree zone, the disagree zone and between them |
| Median, Q1, Q3 | Linear interpolation between ordered ratings (inclusive type 7, as in Excel's QUARTILE.INC and R's default quartile) |
| IQR | Q3 minus Q1 |
| Disagreement index | Optional modified-Delphi spread, (P70 minus P30) / 2.35. Stored on a 1 to 9 scale when there are at least three ratings. It is not the RAND/UCLA disagreement test, and it is not a column of the results table or of the consensus CSV. It affects the class only when the study selects it as the spread measure. |
Conventions:
- The denominator is numeric ratings only. "Unable to rate" is never a score and never part of a percentage, but it is counted and shown. A skipped statement has no answer.
- Retain and Exclude use an integer test, not a rounded percentage. The count in the zone, times 100, must be at least the threshold times the number of numeric ratings. Seven of 10 meets a 70 percent threshold exactly, because 700 is at least 700. Sixteen of 23 does not, because 1600 is not at least 1610, even though 16/23 is 69.57 percent when held to two decimals.
- A display rounded to 70 percent is not that test. The results table rounds to a whole number, so 16 of 23 can be printed as 70 percent. The consensus CSV keeps two decimals and prints 69.57. The class follows the integer test. The difference between the table and the test is an open product gap. It does not mean the threshold was met.
- Type 7 can differ from another quartile convention. The position is
(n − 1) × p, then the value is interpolated between the surrounding ordered ratings. For the synthetic ratings 1, 2, 4, 7, 8, and 9, the first quartile is 2.5, the median is 5.5, the third quartile is 7.75, and the interquartile range is 5.25. The same list in software that uses a different quartile type can produce a different interquartile range. - No mean, standard deviation, or MAD is stored or shown.
Fewer numeric ratings than the study's minimum do not become another class. The classifier returns Too few responses and stops. A methods summary prepared in Setup can still say that too few ratings count as uncertain. Those two sentences are not the same, and the wording difference is unresolved. Do not treat the summary line as the result status.
Zones and scales
A scale is split into a disagree zone, a middle zone and an agree zone, and the shares above are counted against them. The scale and its zones are set on Rating & consensus and lock at launch.
| Scale | Disagree | Middle | Agree |
|---|---|---|---|
| 1–9 (the default for a new study) | 1–3 | 4–6 | 7–9 |
| Likert 1–5 | 1–2 | 3 | 4–5 |
| Likert 1–7 | 1–2 | 3–5 | 6–7 |
| 0–10 | 0–3 | 4–6 | 7–10 |
| Custom | set by the study | set by the study | set by the study |
A new study starts with neutral anchors, such as "Not at all important" and "Critically important", not Retain or Exclude. Panelists are not shown a Status row or the classification label. The block's introduction defines the word Status. That sentence is not a Status row. An older study can still carry anchors that contain decision words. Those anchors are not what a new study starts with.
Classification
Each statement gets one status. The words on screen follow the study's own rule set. The table below is the modified-Delphi set. Under the protocol rule set the words are Consensus in (retain), Consensus out (exclude), No consensus, Subgroup divergence (re-rate), and Deferred for clarity review when too many people are unable to rate. That rule set does not use Polarized as a status, and its group check is not the one in the table.
| Status | When |
|---|---|
| Retain | Every retain condition is met |
| Exclude | Every exclude condition is met |
| Uncertain | Neither set of conditions is met and no other status applies |
| Polarized | Neither retained nor excluded, with ratings clustered at both ends (see below) |
| Group disagreement | Would be retained, but a stakeholder group opposes it (only when the group check is on) |
| Too few responses | Fewer numeric ratings than the study's minimum. Nothing else is decided. |
Retain is checked before Exclude. A statement with too few responses cannot be Polarized.
Starting values
A new study starts from these values. Each one can be changed on Rating & consensus before launch. Setup may describe them as recommended. They are not a universal Delphi standard. The product's minimum of 7 ratings is a classification floor for a statement, not a requirement that every Delphi panel have 7 members.
| Rule | Starting value |
|---|---|
| Retain | at least 70% of ratings in the agree zone, and IQR at most 2 |
| Exclude | at least 70% of ratings in the disagree zone, and IQR at most 2 |
| Minimum responses | 7 numeric ratings |
| Median condition | Off. When it is on, the median must also sit in the agree zone to retain, or in the disagree zone to exclude. |
| Spread measure | IQR. On a 1–9 scale the disagreement index can be used instead. |
These are starting points, not universal rules. A study that departs from them is still the same method, adapted, and its Methods text should say so. The thresholds guide shows every number a study uses and whether it changes an outcome by itself.
Polarized ratings
A median in the middle can hide a panel split between the two ends. With the starting settings, a statement that is neither retained nor excluded is classified Polarized when:
- at least 30% of ratings are in the agree zone,
- at least 30% are in the disagree zone,
- at most 20% are in the middle, and
- at least 6 people rated it.
The check can be switched off. Under the optional protocol rule set, which an organization can turn on, the same pattern is shown as a flag beside the result and never changes it.
The group check
The group check is off by default. When it is on, a statement that would be retained becomes Group disagreement when, in any stakeholder group with enough members (3 by default), more than the allowed share (30% by default) rated it in the disagree zone. This keeps a large group from producing apparent consensus over a smaller group's objection. Groups come from Size & mix, where each is marked core or context.
RAND/UCLA appropriateness
A study using the RAND/UCLA method classifies by the panel median, not by a percentage, and only on a 1–9 scale:
| Result | Rule |
|---|---|
| Appropriate | median 7–9, without disagreement |
| Inappropriate | median 1–3, without disagreement |
| Uncertain | median between the zones, or disagreement |
Disagreement is tested by the interpercentile range adjusted for symmetry (IPRAS) unless the study chooses otherwise. The default constants are IPRr 2.35 and CFA 1.5. Disagreement requires the interpercentile range to be greater than the limit. A range equal to the limit is not disagreement.
A median of exactly 6.5 is uncertain in this product. On the default 1 to 9 scale, appropriate requires a median of at least 7, and inappropriate requires a median of at most 3. A median between those cutoffs is uncertain before disagreement is considered. One published treatment of the RAND/UCLA method calls 6.5 appropriate. This product follows the other treatment, which leaves both 6.5 and 3.5 uncertain. Median 3.5 is uncertain either way. Whether the product should change the 6.5 boundary is not decided here.
The synthetic ratings 6, 6, 6, 6, 7, 7, 7, 7 have median 6.5. The 30th and 70th percentiles are 6 and 7, so the interpercentile range is 1. The default limit for that range is 4.6, and 1 is not greater than 4.6, so IPRAS does not call the panel in disagreement. The class is uncertain because the median is 6.5.
A study can choose the tertile test instead: disagreement when at least a third of the ratings fall in 1 to 3 and at least a third fall in 7 to 9. Equality counts as enough, so three of nine in each extreme zone is disagreement, and three of ten is not. That is a one-third rule. It matches the classic nine-member case and it is not the RAND manual's panel-size table at other sizes, including 10, 13, and 16. Whether to implement that table is open. IPRAS remains the default. The polarized check and the group check do not apply under this method.
Kendall's W
Kendall's W says how similarly the panel ordered the statements in a round. Delphi Studio computes it with the tie correction, from the statements that could be classified and the people who gave a numeric rating to every one of them. Nothing is imputed. Results state W with the number of people and statements it rests on, in these bands:
| W | Described as |
|---|---|
| 0.7 or more | strong agreement |
| 0.5 to under 0.7 | moderate |
| 0.3 to under 0.5 | weak |
| under 0.3 | very weak |
The words in the table are the words on the screen: "strong agreement", "moderate agreement", "weak agreement", and "very weak agreement" in how the panel ordered the statements. The cutoffs are exclusive. 0.7 is strong. A value under 0.7 and at least 0.5 is moderate. They are labels for that sentence, not a validated scale, and this page does not recommend them for a manuscript. No p-value is reported.
The screen sentence uses the stored W, not the two-decimal display. The synthetic value 0.699 is moderate, because it is under 0.7, while two-decimal display rounds it to 0.70.
W is reported and never used by a rule. It is undefined, not zero, when fewer than two people rated every classifiable statement, fewer than two statements could be classified, or everyone gave every statement the same rating. Results then say W could not be worked out, and why. Report W beside the item-level results, not in place of them.
Safeguards around the figures
| Safeguard | What happens |
|---|---|
| Smallest reportable group | A statement answered by fewer people than the Analysis plan's floor (5 by default) is shown as Withheld, with no figures, on every screen and in every file |
| Response-rate notes | At or below 70%, and again below 50%, results and the consensus report show a note. Nothing is blocked. |
| Who was counted | Results always state submitted, counted and left-out counts under the partial-responses rule |
| Comparison with the other partial-responses rule | Results can show how many statements would change class under the other rule. It is labelled "Not the study's rule: for information only" and stores nothing. |
| Rules in force | Each results version records which confirmed rules produced it. On the Results stage, someone who can read Setup can open "Thresholds in force for Round N". Someone who cannot read Setup does not see it. |
What is not computed
- Stability. It can be set on Rating & consensus, but no round measures it yet. It is labelled "Not computed yet", and no result or advice uses it.
- Subgroup comparison. Its settings are kept, but no breakdown or subgroup table is computed, and results do not show one.
- Mean and standard deviation. They are not computed and they are not in the results or the consensus CSV.
- Stopping signals. The only stopping rule is advice set on Rating & consensus: recommend stopping when a set share of statements (90% by default) is retained or excluded. It is labelled advice, is shown with the round plan, and never moves a round.
When the answers behind a result change
What launch freezes, and which numbers can change by a recorded amendment, is Governed study record. A round already rated is never judged again under new numbers.
When a panelist withdraws and asks for their answers to be removed, a round that is not yet finalized is worked out again at once from the remaining answers. For a finalized round the principal investigator decides. Approving writes a new results version from the remaining answers, with the same content and rules, and the earlier version stays on record.
Next steps
- Interpreting results: reading a result and making the decision
- Thresholds: every number in force and what it does
- Reporting: the consensus, participation and response files