Design a defensible modified Delphi study

A defensible Delphi study is not simply a survey sent to people described as experts. It is a governed sequence of independent judgment, controlled feedback, re-rating, and transparent aggregation. The method is most useful when evidence is incomplete, inconsistent, or difficult to apply and a structured account of informed judgment will answer a meaningful research question.

This page explains the methodological decisions a protocol should make before recruitment, and where Delphi Studio records them. It is not a substitute for a study-specific protocol, ethics review, statistical analysis plan, or applicable reporting guideline.

The four defining features

Feature	Methodological purpose	Delphi Studio implementation
Independent responding	Limits status, dominance, and conformity pressure	Panelists respond privately and never see peers' identities or individual responses
Iteration	Gives panelists an opportunity to reconsider judgments	Versioned, multi-round rating workflow
Controlled feedback	Supports reflection without unmoderated debate	Group results from the previous round, filtered by the study's feedback settings, its reporting floor and its comment-sharing choice
Structured aggregation	Makes the group result explicit and reproducible	The study's classification rules, the measures on Consensus analytics, and a supplementary concordance measure

These features are consistent with the classic Delphi model of anonymous response, iteration with controlled feedback, and an aggregated group response described in the RAND literature.

:::note Terminology is not a protocol “Modified Delphi” is an umbrella label, not a complete method specification. Published implementations vary. State exactly what was modified, how initial items were generated, what feedback was provided, how consensus and stability were defined, and why those choices fit the research question. Nasa, Jain, and Juneja provide a useful healthcare-methodology overview. :::

1. Confirm methodological fit

Begin with the decision the study must support—not with a questionnaire.

Good reasons to use Delphi

• Develop recommendations where evidence is limited, inconsistent, or difficult to operationalize.
• Prioritize outcomes, competencies, indicators, research questions, or implementation strategies.
• Refine definitions or classification criteria in an emerging field.
• Identify both areas of agreement and persistent uncertainty.
• Combine evidence with the judgment of people who hold relevant professional or lived expertise.

Reasons to choose another method

Need	Better starting point
Estimate the prevalence of an opinion in a population	Representative survey or probability sample
Answer a question adequately resolved by empirical evidence	Evidence synthesis or primary empirical study
Negotiate interests or jointly solve a problem in real time	Facilitated workshop, nominal group, or deliberative process
Validate a predetermined conclusion	Reframe the project; consensus must not be used as ceremonial endorsement

Write down the intended output before creating items. Examples include “define a minimum outcome set,” “prioritize implementation barriers,” and “map unresolved disagreement.” In Delphi Studio, the Study design steps of Setup record the study's identity, its ethics approval and data governance, and its method.

2. Define what “modified” means

Traditional and modified Delphi studies often share the same core safeguards but differ in how the first item set is produced.

Design feature	Traditional Delphi	Modified Delphi
Starting point	Often begins with open idea generation	Often begins with candidate items prepared before Round 1
Common item sources	Panelist proposals	Reviews, prior studies, interviews, frameworks, or steering-group work
Typical emphasis	Exploration and concept generation	Refinement, prioritization, validation, or appropriateness
Reporting obligation	Explain how open responses became rateable items	Explain item sources, steering-group influence, and every other modification

Starting from an evidence-informed item bank can reduce burden, but efficiency alone is not a methodological justification. Report how the item set was assembled, piloted, edited, and checked for missing perspectives.

3. Engineer the protocol before recruitment

The protocol should make consequential decisions before anyone sees results. Specify the study objective, who is eligible for the panel, what each rating means, the consensus rule, and the stopping rule. State why each choice fits the question. A later reader should be able to tell a planned rule from a change made after results existed.

Define the construct and scale

State what every rating means: importance, agreement, feasibility, appropriateness, inclusion, or another construct. Define and report the anchors for every scale point or category. A 1–9 importance scale and a 5-point agreement scale are different measurement designs, even if both ultimately produce an “include” decision.

In Delphi Studio, the Rating & consensus step of Round design can name up to five rating dimensions and mark one of them primary. The answer stored for a statement is one rating. The primary dimension is the one the consensus rule uses. The same step sets the scale, the anchors panelists read, the zones, and the decision rules.

Pre-specify consensus and item handling

There is no universal Delphi consensus threshold. Select a rule that fits the construct, panel, and intended use; justify it; and state how missing or unable-to-rate responses enter the denominator.

For example, a protocol might define inclusion on a 9-point scale as:

• at least 75% of valid ratings in the high zone (7–9); and
• no more than 15% in the low zone (1–3).

Example, not a starting value

The 75/15 pair is an illustration of a protocol choice. It is not the value a new Delphi Studio study starts from. A new study starts from seeds: 70 percent in the agree zone, 70 percent in the disagree zone, an interquartile-range ceiling of 2, and at least 7 numeric ratings before a statement is classified. The minimum of 7 is the number of ratings required to classify one statement. It is not a recommended panel size. Those seeds are starting values a study can edit. They are not a universal Delphi standard. See Consensus analytics and Thresholds.

Also specify what happens to each result. On Modified Delphi: percentage and spread, the result names are:

• Retain, when the inclusion rule passes;
• Exclude, when the exclusion rule passes;
• Uncertain, when neither rule passes;
• Polarized, when the study's opposed-cluster rule passes;
• Group disagreement, when a counted group opposes a statement that would otherwise be retained and the study has turned that check on; and
• Too few responses, when there are fewer numeric ratings than the minimum.

Before the last round, Uncertain, Polarized, Group disagreement, and Too few responses go to Advance, and the statement continues as written. The last round is different. Rewording is not one of those result classes. It is a clarity-review choice, "Reword it" (Revise). See Interpreting results.

Pre-specify stopping and stability

Maximum rounds and consensus are not the same as stability. A panel may remain stably divided, which is a meaningful result rather than a failed study.

Define stability in the protocol, for example as a permitted change in the median or the interquartile range between rounds. Delphi Studio lets a study record its stability settings, but it does not compute stability, and its screens say "Not computed yet". It does not measure a change in the median or the interquartile range. It does record stopping advice: consider stopping once a stated share of statements (90% by default) is settled, meaning retained or excluded. That advice can be amended after launch, with a reason. It does not move a round by itself, and it does not decide that the study has reached enough consensus to end. The research team remains responsible for the decision to stop and its rationale. See Consensus analytics.

Govern changes and overrides

Not every field becomes immutable at launch, and not every field can still be changed. After launch, the scale, the zones, the consensus method, and the spread rule stay fixed. On Modified Delphi: percentage and spread, the consensus numbers can change by a recorded amendment from the next round that has not opened. That amendment is refused for RAND/UCLA and for the protocol rule set. RAND/UCLA also refuses a later change to the disagreement test and the IPRAS constants. Stopping advice can still be amended with a reason. That change does not reclassify a round already rated. The full boundary is Launch and amendments.

Guarding against rule drift

Results versions and the audit log make a later change visible and keep the rules a rated round used. They reduce the risk of moving a threshold after seeing results. They do not replace investigator judgment, protocol discipline, or transparent reporting.

4. Engineer the panel around relevant expertise

Reputation is not a substitute for fit. Define the knowledge, experience, and perspectives required to answer the consensus question, then recruit against those criteria rather than against known positions on the topic.

An expertise-and-perspective matrix can make those choices reviewable. This example is illustrative rather than a platform default:

Perspective	Why it is needed	Illustrative eligibility evidence	Illustrative target
Patient or caregiver	Identify outcomes that matter in daily recovery	Direct relevant lived experience	At least 20% of completing panel
Clinician	Judge clinical relevance and interpretability	At least three years of relevant practice	Balance across relevant disciplines
Outcomes researcher	Assess construct and measurement clarity	Relevant methods or core-outcome work	At least three completing members
Implementation leader	Judge operational feasibility	Deployment or quality-design responsibility	At least two completing members

A narrowly technical question may appropriately require a more homogeneous panel. What matters is that the planned composition follows from the question and that omissions are acknowledged. The size of the panel is a recruitment plan. It is separate from the minimum number of ratings required before one statement is classified.

Track:

• how candidates were identified;
• eligibility and exclusion criteria;
• numbers screened, invited, enrolled, and completing each round;
• relevant disciplines, regions, roles, and experience;
• reasons for non-participation or withdrawal when known and appropriate to report; and
• any recruitment deviation or steering-group judgment.

Delphi Studio records study participation in the study itself (see Recruitment and consent). The expert directory keeps each person's answer about future contact separate from any study's consent. A recorded refusal blocks adding that person. Someone who has not been asked is not treated as having agreed.

Response anonymity

Panelists never see peer identities or peer-level responses. Staff screens show pseudonymous codes. People being recruited can be shown by name on the recruiting screens. Revealing an enrolled person's name takes a written reason and is audited. A coordinator can only request a reveal; a principal investigator or an administrator approves it. See Identity and pseudonyms.

5. Build judgeable items

Each item should express one proposition in language that the intended panel can interpret consistently.

Weak item	Problem	Stronger alternative
“Remote monitoring is effective, acceptable, and should be routinely implemented.”	Three claims in one rating	“Patient acceptability should be included in the minimum outcome set.”

Document the source of every initial item. Pilot the instructions, anchors, item wording, unable-to-rate option, and expected completion time with people similar to the intended panel. Piloting is a distinct methodological activity and should not be silently merged into the formal first round.

Delphi Studio's Item bank can record a source as a DOI, a PMID, a URL, or free text. A source is not required on every statement. A statement keeps an id such as S07. A later wording is shown as Version 2. When a rating round has already sent the statement, a revision can cite that round's stored results. Later reports can then explain what changed, when, and why. Rewording does not by itself make the new wording directly comparable with the earlier ratings.

6. Run iterative rounds without steering the panel

The purpose of feedback is reflection, not forced convergence. The software does not close a study because the ratings have converged.

For Round 2 and later, useful feedback may contain:

• the panelist's own previous rating;
• the group distribution, median, and interquartile range;
• zone percentages or another protocol-specified summary;
• approved, de-identified comments or balanced themes; and
• a clear indication when wording changed between rounds.

In Delphi Studio, Round design chooses what live feedback can include: the median, the middle half of the ratings, the zone percentages, and the person's own earlier rating. Setup has a "Show a histogram" switch, and the Setup preview draws bars. The live feedback block does not draw a histogram, and it does not use the label IQR. It can show "Middle half of ratings". Comments are shared only as the recorded comment-sharing choice allows, and statements below the reporting floor are withheld. Feedback shows the previous round's current verified results and is not pinned when the round opens. Each panelist sees the same group evidence plus their own earlier rating. See Rounds and For panelists.

Interpret more than the median

Concept	Question answered	Typical evidence
Location	Where are ratings concentrated?	Median and zone percentages
Dispersion	How spread out are ratings?	Interquartile range, and on a 1–9 scale the disagreement index. A protocol may also plan a distribution display.
Item-level consensus	Did the pre-specified classification rule pass?	Each rule's check, shown beside the result
Concordance	Do panelists rank the set of items similarly?	Kendall's W, when applicable
Stability	Are item summaries still changing between rounds?	A change defined in the protocol. Delphi Studio does not compute it, and the screen says "Not computed yet".
Polarization	Is a central summary hiding opposed groups?	The share of ratings in each extreme zone. A separate subgroup comparison is not computed.

Kendall's W is supplementary. Delphi Studio computes it with a tie correction, from the statements that could be classified and the people who rated every one of those statements, and uses it in no rule. It must not replace the item-level consensus definition. The display bands are screen labels. They are not a validated interpretation, and this page does not recommend them for a manuscript.

Code comments with independent coders

Panelists' comments carry the reasons behind ratings, including conditional positions and dissent. In Delphi Studio, two coders code comments blind to each other and an adjudicator settles their differences. Pooled kappa is reported as a quality indicator and never stops the study. Researchers remain responsible for checking coverage, preserving meaningful dissent, and documenting their decisions. See Coding and adjudication.

7. Stop transparently and report the full process

Do not report only the final list. Readers need enough information to understand who contributed, how the items and rules evolved, what feedback was provided, and where uncertainty remained.

For biomedical consensus research, use ACCORD as the primary reporting checklist. ACCORD applies across biomedical consensus methods but is explicitly a reporting guideline, not a conduct standard. CREDES offers complementary Delphi-specific conduct and reporting guidance developed in palliative care. Neither makes the software compliant with the guideline. See the ACCORD and CREDES crosswalk.

At minimum, report:

• the objective, intended users, and reason for choosing Delphi;
• the exact modification and initial item sources;
• ethics review or determination;
• steering-group roles and conflicts of interest;
• panel eligibility, recruitment, composition, and attrition;
• piloting and survey materials;
• every round's dates, participation, feedback, and results;
• consensus, item-handling, stability, and stopping rules;
• item revisions, overrides, amendments, and deviations; and
• limitations, unresolved disagreement, funding, and applicability.

Delphi Studio's reports and files come from the stored results versions, the decision log and the study's records. It provides a consensus report and a participation report on screen; consensus, participation and responses CSV files for the study or one round; a counts-only study summary in Markdown with an outline CSV; and a study audit CSV. Study-report files omit panelist names and email addresses, while two principal-investigator downloads can contain a name and an email address and the organization audit download can contain a staff member's display name; the reporting floor, 5 by default, is not the classification minimum of 7. See Reporting and Identity and pseudonyms.

Failure-mode diagnostic

Failure	Why it threatens quality	Remedy
Post-hoc thresholds	Makes classification vulnerable to outcome-driven interpretation	Pre-specify rules and record any later change before it is applied
Reporting only medians	Can hide dispersion and polarization	Report distributions, the interquartile range, and polarization or group checks where planned
Forced convergence	Confuses fatigue or conformity with agreement	Use stopping rules and report stable disagreement
Overwriting revised items	Erases what panelists actually rated	Create versioned items with rationale and lineage
Selective feedback	Can steer later-round ratings	Use a declared feedback policy and preserve opposing or conditional positions
Undefined expertise	Prevents readers from judging panel relevance	Pre-specify eligibility and report composition
Silent attrition	Makes later-round results difficult to interpret	Report denominators and participation flow for every round
Unreported steering decisions	Conceals investigator influence	Record rationale, timing, and actor for overrides and revisions

Pre-launch review

• The objective requires structured judgment rather than a representative population estimate.
• The exact Delphi modification and item sources are documented.
• Panel eligibility and perspective targets follow from the research question.
• Every scale has explicit anchors and an unable-to-rate policy.
• Consensus, polarization, item-handling, stability, and stopping rules are pre-specified.
• Piloting is distinguished from formal rounds.
• Feedback content and approval procedures are declared.
• Planned amendments, overrides, and deviations require rationale and audit evidence.
• ACCORD reporting fields and applicable CREDES elements have an evidence source.
• The research team has reviewed the protocol on Review & launch before launch.

Further reading

• Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology. 2021;11(4):116–129.
• Gattrell WT, et al. ACCORD: A reporting guideline for consensus methods in biomedicine. PLOS Medicine. 2024;21(1):e1004326.
• Logullo P, et al. ACCORD explanation and elaboration. PLOS Medicine. 2024;21(5):e1004390.
• Jünger S, et al. Guidance on Conducting and REporting DElphi Studies (CREDES). Palliative Medicine. 2017;31(8):684–706.