The Work Between Rounds

Much of the methodological risk in a Delphi study sits in the interval between closing one round and opening the next.

During a round, panelists rate and the study records. Between rounds, investigators make judgment calls: which items advance, how group opinion is summarized, whether wording changes, and how free-text comments are used. These decisions can substantially shape the next round's ratings, yet they are often among the least visible parts of the published method.

Feedback is an intervention

Whatever you show panelists can change what they do next. That is the point—controlled feedback is the iterative mechanism of the method. It also means the feedback design belongs in the protocol, not in an ad hoc email written after Round 1.

Decide in advance and apply the policy consistently:

• Group statistics. Median and IQR, percentages by response region, or a full histogram. A distribution can reveal disagreement that central tendency alone conceals.
• The panelist's own prior rating. Showing it supports deliberate reconsideration; omitting it encourages a fresh rating. Either choice should be reported.
• Stratum-level breakouts. These can reveal systematic differences, but small cells may expose identities or overstate unstable percentages.
• Free-text handling. Pre-specify whether comments will be coded, summarized, sampled, or shown verbatim. Document selection and redaction, and protect confidentiality. “Show everything” is not safe when comments contain identifying or sensitive information; “show a convenient subset” is not methodologically transparent.

The ACCORD development study provides a concrete example: panelists received category-level voting distributions and relevant anonymized comments, wording changes were recorded, and materially altered statements were treated as new items.

Revise items sparingly and version them always

Free text may reveal that an item is ambiguous, double-barreled, or interpreted differently across specialties. Clarifying it can improve validity. But revisions change what panelists are judging, so the analysis must preserve the relationship between versions instead of overwriting the original text.

Practical discipline:

• Record the rationale for every change and, where relevant, the comments or investigator decision that prompted it.
• Create a new version with a link to the prior wording; never overwrite the text in place.
• Tell panelists when wording changed and distinguish minor clarification from a change in meaning.
• Treat a materially altered proposition as a new analytic item, or explain and justify a different rule in the protocol.
• Report the number and type of revisions so round tables reconcile.

This prevents a subtle failure mode: repeated rewording until an item crosses the threshold. Each edit may appear responsive, but the final proposition may never have received the same iterative exposure as the original.

New items need a policy

Panelists may propose items the investigators omitted. Accepting them can be one of the method's strengths, but a late-added item has fewer rounds of exposure than the original set. State whether new suggestions are permitted, the last round in which they can enter, how they are screened, how many rating rounds they require, and what threshold applies.

Track stability, not just consensus

Convergence, consensus, and stability are related but different. An item can remain at roughly the same divided distribution across multiple rounds. That stable disagreement is a finding, not necessarily a reason to keep running rounds until fatigue produces movement.

If stability informs stopping, define it numerically before data collection—for example, a maximum permitted round-to-round change in median and IQR—and apply it only to items observed in both rounds. Report stable contested items separately from items that reached positive or negative consensus.

What the platform does

Delphi Studio creates a linked item version for each revision and requires a rationale, with optional source-comment references. The next-round plan records which revised, re-rated, retained, dropped, or newly proposed items move forward. Feedback packets can be locked before a later round opens; otherwise, each panelist's packet freezes on first view. Per-round stability metrics and stopping recommendations remain advisory rather than silently changing dispositions. See Running rounds for the full workflow.

Next post: turning the record into a manuscript—reporting standards, participant flow, and the supplement reviewers can actually use.