The Work Between Rounds

Much of the methodological risk in a Delphi study sits in the interval between closing one round and opening the next.

During a round, panelists rate and the study records. Between rounds, investigators make judgment calls: which items advance, how group opinion is summarized, whether wording changes, and how free-text comments are used. These decisions can substantially shape the next round's ratings, yet they are often among the least visible parts of the published method.

Feedback is an intervention

Whatever you show panelists can change what they do next. That is the point—controlled feedback is the iterative mechanism of the method. It also means the feedback design belongs in the protocol, not in an ad hoc email written after Round 1.

Decide in advance and apply the policy consistently:

• Group statistics. Median and IQR, percentages by response region, or a full histogram. A distribution can reveal disagreement that central tendency alone conceals. A protocol may plan to show a histogram. Delphi Studio's live feedback block does not draw one. It shows words and numbers. Show a histogram on Round design, and the bars in the setup preview, are not that block.
• The panelist's own prior rating. Showing it supports deliberate reconsideration; omitting it encourages a fresh rating. Either choice should be reported.
• Stratum-level breakouts. These can reveal systematic differences, but small cells may expose identities or overstate unstable percentages. Delphi Studio's subgroup comparison is not computed.
• Free-text handling. Pre-specify whether comments will be coded, summarized, sampled, or shown verbatim. Document selection and redaction, and protect confidentiality. “Show everything” is not safe when comments contain identifying or sensitive information; “show a convenient subset” is not methodologically transparent.

The ACCORD development study provides a concrete example: panelists received category-level voting distributions and relevant anonymized comments, wording changes were recorded, and materially altered statements were treated as new items.

Revise items sparingly and version them always

Free text may reveal that an item is ambiguous, double-barreled, or interpreted differently across specialties. Clarifying it can improve validity. But revisions change what panelists are judging, so the analysis must preserve the relationship between versions instead of overwriting the original text.

Practical discipline:

• Record the rationale for every change and, where relevant, the comments or investigator decision that prompted it.
• Create a new version with a link to the prior wording; never overwrite the text in place.
• Tell panelists when wording changed and distinguish minor clarification from a change in meaning.
• Treat a materially altered proposition as a new analytic item, or explain and justify a different rule in the protocol.
• Report the number and type of revisions so round tables reconcile.

This prevents a subtle failure mode: repeated rewording until an item crosses the threshold. Each edit may appear responsive, but the final proposition may never have received the same iterative exposure as the original.

New items need a policy

Panelists may propose items the investigators omitted. Accepting them can be one of the method's strengths, but a late-added item has fewer rounds of exposure than the original set. State whether new suggestions are permitted, the last round in which they can enter, how they are screened, how many rating rounds they require, and what threshold applies.

Track stability, not just consensus

Convergence, consensus, and stability are related but different. An item can remain at roughly the same divided distribution across multiple rounds. That stable disagreement is a finding, not necessarily a reason to keep running rounds until fatigue produces movement.

If stability informs stopping, define it numerically before data collection—for example, a maximum permitted round-to-round change in median and IQR—and apply it only to items observed in both rounds. That numeric rule is an illustration of a protocol choice, not a result Delphi Studio calculates. Report stable contested items separately from items that reached positive or negative consensus. Stopping advice does not change a disposition.

What the platform does

In Delphi Studio a revision keeps the original wording: the statement keeps its permanent ID, the new wording becomes its next version, and the revision cites the results behind it. While another round follows, the Decision stage uses three dispositions: Advance (rated again as written), Revise (the clarity choice to reword), and Hold (settled, not rated again). On the last planned round the names are the five final results, not those three. To reword is not carried when someone confirms the plan. The labels, and what a reword withholds, are on Interpreting results. Overriding what the rules imply needs a written reason that goes into the decision log. Panelist suggestions are off by default and never change a round in progress. Comments are shared only as the study's recorded comment-sharing choice allows. Feedback is built from the previous round's current results; it is not frozen when a round opens. Stability is not computed yet, and the stopping recommendation is advice only: it never changes a disposition. See Rounds for the round sequence.

Next post: turning the record into a manuscript—reporting standards, participant flow, and the supplement reviewers can actually use.