Interpreting results
After collection closes, someone computes the results. The round's Results page then shows the study's classification for every statement. A person confirms what each statement does next. The product does not confirm that plan by itself. This page is the map from a class to that decision. The sequence a round goes through is in Rounds.
Five layers stay distinct. A statistical result is a figure, such as a median or a zone share. An item classification is the badge the study's rules assign. A researcher decision is the disposition a person confirms. A plan tile is the group on the Decision stage. A next-round action is what that confirmation does to the Item bank.
Reading the Results page
The top of the page says which results version you are reading, when it was computed, from how many submitted responses and ratings, and under which rules (a short fingerprint of the rules in force for the round). Under that:
- Who was counted. One sentence gives how many people submitted, how many were counted, and how many the partial-responses rule left out and why.
- A response-rate note, when the rate is at or below the Analysis plan's warning level. It is a note. It blocks nothing.
- Counts by outcome, in the study's own words, with a separate count of withheld statements.
- Kendall's W, in one sentence with the people and statements it rests on. It is within-round concordance: how far the panel's orderings of the statements agree. It is reported for the analyst and not used by any rule. It is not an item class and not stability. No p-value is computed.
The table lists each statement with its permanent ID, grouped by area:
| Column | What it shows |
|---|---|
| ID | The statement's permanent ID, for example S07 |
| Statement | The wording that was rated |
| Result | The classification, and one plain sentence saying why, for example "Every retain condition is met." |
| Agree zone | Share of numeric ratings in the agree zone, headed with the zone, for example "Rated 7–9" |
| Disagree zone | Share in the disagree zone |
| Median and IQR | Centre and spread |
| Answered | Numeric ratings, and how many answered "Unable to rate" |
A statement answered by fewer people than the Analysis plan's smallest reportable group shows Withheld and no figures. On a phone the same rows are shown as cards.
The class does not use the rounded percent in that table. It compares the count times 100 with the threshold times the number of numeric ratings. 16 times 100 is 1600 and 70 times 23 is 1610, so 16 of 23 fails a 70 percent threshold. The table rounds the printed share to the nearest percent, so 16 of 23 can appear as 70 percent. The consensus CSV keeps two decimal places (69.57). That display gap is open. This page does not choose a rounding rule. The measures are on Consensus analytics.
The page also has a folded Thresholds in force table for the round, and, before the round is finalized, a comparison with the other partial-responses rule. That comparison is labelled "Not the study's rule: for information only".
Staff Results shows the badge for the study's rule set. The rating screen's Preview tab can show the same badge for sample ratings. Panelist feedback does not show that badge, and it does not show a Status row.
Reading a single statement
These examples use a 1–9 scale and the figures a new study starts with: retain at 70 percent in 7–9 with an interquartile range of at most 2, exclude at 70 percent in 1–3 with an interquartile range of at most 2, at least 7 numeric ratings, and the polarized check on at 30 percent in each extreme zone, at most 20 percent in the middle, from at least 6 ratings. Those figures are seeds. They are not a standard this page recommends, and it does not say whether a study should adopt them. Each example has ten ratings unless stated. The interquartile range is the type-7 range. The tables are classes only. They are not the decision.
Retain. Ratings 5, 6, 7, 7, 8, 8, 8, 9, 9, 9.
| Measure | Value |
|---|---|
| Rated 7–9 | 80% (8 of 10) |
| Median | 8 |
| IQR | 1.75 (Q1 7, Q3 8.75) |
| Result | Retain: every retain condition is met |
Polarized. Ratings 1, 1, 2, 2, 3, 5, 7, 8, 9, 9.
| Measure | Value |
|---|---|
| Rated 7–9 | 40% |
| Rated 1–3 | 50% |
| Middle (4–6) | 10% |
| Median | 4 |
| Result | Polarized: neither retained nor excluded, with both ends at 30% or more and at most 20% in the middle |
The median alone would suggest a lukewarm panel. The split is the finding. On the modified rule set, Polarized is a status. On the protocol rule set it is a flag, Polarized pattern, shown for information., and it does not change the status. The protocol badges are Consensus in (retain), Consensus out (exclude), No consensus, and Subgroup divergence (re-rate). That shorter No consensus is a classification badge. It is not the last-round plan tile.
Uncertain. Ratings 4, 5, 5, 6, 6, 6, 7, 7, 8, 8.
| Measure | Value |
|---|---|
| Rated 7–9 | 40% |
| Median | 6 |
| IQR | 1.75 |
| Result | Uncertain: neither the retain nor the exclude conditions are all met |
A small spread is not agreement by itself. Here the panel is close together in the middle of the scale.
Too few responses. Six numeric ratings, whatever they are, are Too few responses when the minimum is 7. No other class is assigned. The methods text can still say that fewer ratings count as uncertain. That sentence and the badge disagree, and this page does not settle which wording a manuscript should prefer.
A RAND/UCLA study uses a different badge on Results and on that Preview tab: Appropriate, Inappropriate, Uncertain, or Too few responses. Whether a median of 6.5, or RAND Table 4 rather than the tertile test, should be used is open. This page does not add a 6.5 example. See RAND/UCLA.
From a result to the next step
While another round exists, Uncertain, Polarized, Group disagreement, and Too few responses are one outcome. The plan tile reads Rated again. The disposition reads Advance. Those are two labels for that outcome, not two actions, and not a reword. Reword it is a separate clarity choice. It leads to the tile To reword and the disposition Revise.
On the last planned round the Decision stage does not show a Rated again tile. A statement that would have been rated again, and that did not come from a clarity review, is in Final: no consensus. A clarity review that kept the wording or chose a reword, with no round left, is Final: deferred, still unclear. The shorter string No consensus is not that tile. An override line uses it when no next round exists. The protocol rule set uses it as a classification badge. Those are labels for an override line and for a protocol badge.
While another round exists, the plan tiles are:
| Plan tile | Disposition | What it is |
|---|---|---|
| Retained | Hold | Consensus in. Not rated again. |
| Excluded | Hold, after any steering review | Consensus out. An excluded statement with a steering flag has no disposition until that review is recorded. |
| Rated again | Advance | Uncertain, Polarized, Group disagreement, or Too few responses. Also Keep it as written and Send it back for re-rating. The wording is unchanged. |
| To reword | Revise | The clarity choice Reword it. Not in the next round until someone writes the new wording in the Item bank and assigns it. |
| Retired | Hold | The clarity choice Retire it. Dropped, and kept in the Item bank with the reason. |
| Waiting | None yet | The section heading is Waiting for a review decision. A pending review blocks confirmation. |
Hold covers Retained, Excluded, and Retired. Hold means settled: not rated again.
Study team offers two labels for who confirms what the rules decided: The principal investigator or Biostatistics, and Biostatistics only. A new empty team record starts on the first. That is not a default of Biostatistics only. An older record that omits the field is read as Biostatistics only. The protocol rule set keeps confirming apart from the principal investigator, so only the biostatistics role confirms what the rules decided. A coordinator cannot open Results and cannot confirm the plan.
Confirmation retires Retired statements. When a next round exists, it assigns only Rated again statements. It does not write new wording, and it does not carry To reword.
The control is Confirm Round N's decisions. The dialog title is Confirm Round N's decisions?. The confirm control is Confirm decisions. When a statement went to a steering review, the steering committee uses Confirm the escalated dispositions before anyone can confirm the plan. That step is a prerequisite. It is not a second copy of the plan.
Reviews under the protocol rule set
A study using the optional protocol rule set has two review queues. A statement in a queue has no disposition until it is decided, and each decision needs a written reason. Only the biostatistics role can decide a clarity review. Only the steering committee can decide a steering review. The principal investigator cannot decide either queue.
Clarity review. Raised when too many panelists could not rate the statement.
- Reword it leads to To reword / Revise.
- Keep it as written leads to Rated again / Advance.
- Retire it leads to Retired / Hold.
Steering review. Raised when an excluded statement was opposed by a counted group.
- Confirm consensus out leads to Excluded / Hold.
- Send it back for re-rating leads to Rated again / Advance.
Overrides
There is no free edit of a rule outcome. An override is a review decision that leads somewhere other than where the rule's classification points: the steering committee sending a consensus-out statement back for re-rating, or the biostatistics role retiring a deferred statement.
- Its reason must be at least 20 characters. A name the check is sure of is refused. A single word that only might be a name is a warning, not a refusal.
- The person confirming the plan ticks each override.
- Each override is written to the decision log before the plan is confirmed, and shown with the rule outcome, the decision, and the reason.
Confirming a round's decisions is final. The decision log only grows, and each entry seals the one before it, so a change to an earlier entry is detectable. Changing a finalized round's results needs an authorized amendment or an approved answer removal, and each of those records a new results version beside the old one.
Changing wording between rounds
The next round's statements can be edited until its register is locked. A revision of a statement that a rating round has sent cites that round's stored result, so a reader can see what the change was based on. Confirmation does not carry that revision. To reword stays out until the new wording is written in the Item bank and assigned.
After the last round
On the last round, the Decision stage groups statements under five final-result labels. It does not show a Rated again tile. Final: no consensus is that group. The shorter No consensus is not.
| Final result | Meaning |
|---|---|
| Final: retained | consensus in |
| Final: excluded | consensus out |
| Final: retired | dropped by a review |
| Final: no consensus | would have been rated again, did not come from a clarity review, and no round is left |
| Final: deferred, still unclear | too many panelists could not rate it, the clarity review kept or reworded it, and no round was left |
If statements were left to rate again, the principal investigator can add one further rating round as a recorded amendment with a reason. The plan confirmed at launch stays as it was, and the study summary shows how many rounds were planned and how many ran.
Report every round, not only the final list. Final: no consensus is a result, not a failure of the panel.
When to stop
Rating & consensus can hold a stopping rule: recommend stopping when a set share of statements (90 percent in the new-study seed) is retained or excluded. It is advice. It is shown with the round plan and never moves a disposition. The decision to stop, or to run the planned rounds, belongs to the team, and the manuscript should say who made it and why.
Round-over-round stability and the subgroup comparison are labelled Not computed yet. You can record stability settings. The product does not compute stability. A rating mean and a standard deviation are not computed on Results. The Analysis plan can still name Mean and SD as a summary choice. That choice is not a computed result. Trajectory views and a publication checklist are not produced here.
Between rounds
A typical sequence after reviewing results:
- Decide any clarity and steering reviews, each with a reason. Until that decision, the statement is Waiting and the plan cannot be confirmed.
- The person who may confirm uses Confirm Round N's decisions. When a steering review exists, the steering committee confirms the escalated dispositions first. A coordinator does not take this step. Confirmation retires Retired statements and, when another round exists, assigns only Rated again statements.
- Where the tile is To reword, write the new wording in the Item bank and assign it. Confirmation does not do that.
- Choose which comments panelists see in the next round. If you choose nothing, panelists see no comments.
- Prepare the next round, which records its statements. Locking the register, approving the round, and opening it are on Rounds. This page does not repeat those steps.
After a reword, the live feedback block withholds the earlier group figures, the person's earlier rating, and the comments the team had shared. The panelist reads that the statement was reworded after that round, so the earlier numbers are not shown. When the earlier figures are shown, the block is words and numbers: the median, Middle half of ratings, and the zone shares, including the middle share labelled In between, plus the person's own earlier rating and the comments the team chose. It does not print a row labelled IQR, it does not draw a histogram, and it does not show why the wording changed. The block's introduction still defines the word Status. That sentence is not a Status row, and the block does not show Appropriate, Inappropriate, Retain, or Exclude. A new study starts with scale anchors that do not use Retain or Exclude. A study that kept the older anchors can still show those words beside the numbers. Show a histogram on Round design, and the bars in the setup preview, are a different screen.
Group feedback uses the previous round's current verified results. It is not pinned when the round opens: the page says it shows the current results. Results based on fewer people than the study's smallest reportable group are withheld.
Next steps
- Consensus analytics: the measures and the rules
- Rounds: the sequence a round goes through, from approval to the next round
- Decision: the dispositions and the final results
- Multi-round worked example: an end-to-end scenario that follows this page. An unsettled class is Advance and Rated again while another round remains. The example does not assign Revise to that class, draw a histogram, or show group figures after a reword.
- Reporting: the consensus report and files