Modified e-Delphi
Modified Delphi is a family of Delphi designs, not one protocol. Experts rate statements on their own, comment where the design asks, see controlled group feedback, and may rate again in a later round. A statement can meet the study's rule, remain uncertain, or show disagreement. The planned rounds can also run out before every statement is settled. When that design is run on the web, it is often called a modified e-Delphi. In Delphi Studio, electronic delivery is how the study is conducted. It is not a separate scientific method, and it is not a separate preset.
The Method step marks Modified Delphi as the recommended preset. That is a product label. It is not a requirement that every study use the preset, and the label does not state what this study modified. Say that in the protocol.
Delphi Studio records the workflow. You confirm the scale, zones, and consensus rules in Setup before collection. What launch freezes, and what a recorded amendment can still change, is in Launch and amendments. Each round opens only after the principal investigator approves its content, rules, consent form, and schedule. Results are computed the same way every time and stored as sealed versions. See Consensus analytics. Every statement keeps a permanent ID through its revisions. You remain responsible for revising statements, choosing what feedback panelists see, recording overrides, and deciding when to stop. This page explains the design. It is not the procedure. The procedure starts at Start a new study, which creates a draft.
When to use modified e-Delphi
Use a modified Delphi design when the question is whether experts agree that a statement is important, accurate, appropriate for inclusion, or clearly worded. It can suit competency frameworks, quality indicator sets, curriculum outcomes, and policy priorities. Agreement is not guaranteed.
| Use case | Example item |
|---|---|
| Competency frameworks | "A rural telehealth nurse should triage acute chest pain via video within 5 minutes." |
| Quality indicators | "Monthly caseload review is essential for safe community mental health practice." |
| Curriculum outcomes | "Graduates should interpret basic ECG strips without supervision." |
| Policy priorities | "Funding should prioritize preventive care over acute expansion in Year 1." |
Choose this design rather than the RAND/UCLA Appropriateness Method when the task is agreement with normative or definitional statements, rather than the appropriateness of clinical actions under defined indications. When the panel should write the items first, the Classical Delphi choice on Method pairs an open first round with later rating rounds. See Comparing Delphi types.
Starting values on a new study
A new study starts from the values below. They are editable before launch. They are seeds, not the definition of the method and not a universal standard. This page does not say whether a study should adopt them. Change them in Setup before launch when the protocol needs a different design, and say so in the Methods text. What launch freezes is in Launch and amendments.
| Setting | Starting value | Notes |
|---|---|---|
| Scale | 1–9 | The workspace default for new studies. Likert 1–5, Likert 1–7, 0–10, and custom scales are also offered. The scale is set on Rating & consensus. Choosing a method does not change it. |
| Zones on 1–9 | Disagree 1–3, uncertain 4–6, agree 7–9 | Each scale has its own preset zones, editable before launch. |
| Retain | At least 70% of numeric ratings in the agree zone and IQR at most 2 | Both conditions must hold. This pair is a seed. |
| Exclude | At least 70% in the disagree zone and IQR at most 2 | Symmetric to Retain. This pair is a seed. |
| Minimum responses | 7 numeric ratings | Below this the status is "Too few responses" and nothing else is decided. The 7 is a seed, not a required panel size. |
| Median condition | Off | Can be switched on, requiring the median to sit in the agree zone to retain, or in the disagree zone to exclude. |
| Spread measure | IQR | On a 1–9 scale the disagreement index can be selected instead. That index is not the RAND/UCLA disagreement test. |
| Polarized | On: at least 30% in each extreme zone, at most 20% in the middle, from 6 ratings | A statement that is neither retained nor excluded but matches this pattern is classified Polarized. These figures are seeds. |
| Planned rounds | 3 for the three named workflows | "Rate existing items", "Generate items, then rate", and "Define and refine, then rate" start from 3 rounds. "Build a custom workflow" starts from one rating round. The count is a seed, not a standard. |
| Stopping advice | Suggest stopping when at least 90% of statements are settled | Advice only. Nothing moves a round. |
| Primary dimension | Importance | Up to five dimensions can be rated. Only the primary one decides consensus. |
| Panelist suggestions | Off | A per-round switch. Suggestions never change the round they are made in. |
Stability between rounds and a subgroup comparison can be configured. Delphi Studio does not compute them. Setup labels them "Not computed yet". The Thresholds table on Review & launch lists every number in force and whether it is enforced, reported only, advice only, not applied yet, or off.
The rating scale and zones
Panelists rate each statement on the study's scale and may choose Unable to rate when the statement is outside their expertise. That choice is on by default. Unable-to-rate answers are left out of the denominator for the median, the interquartile range, and zone percentages. Their count is still shown with each result.
By default a comment is asked for when the rating is low. The study can instead never ask, or ask for low ratings and for Unable to rate. Comments feed comment coding.
Panelists read neutral anchors next to the numbers. A new study proposes "Not at all important", "Uncertain or depends on context", and "Critically important". You can edit the anchors before launch. Anchors on a new study do not say Retain or Exclude. The live feedback block does not show a Status row or the classification label.
Preset zones
| Scale | Disagree | Uncertain | Agree |
|---|---|---|---|
| 1–9 | 1–3 | 4–6 | 7–9 |
| Likert 1–5 | 1–2 | 3 | 4–5 |
| Likert 1–7 | 1–2 | 3–5 | 6–7 |
| 0–10 | 0–3 | 4–6 | 7–10 |
How a statement is classified
The rules are checked in a fixed order:
- Fewer than the minimum numeric ratings: Too few responses.
- Every Retain condition met: Retain. If the optional group check is on and a large enough group opposes the statement beyond the set share, the result is Group disagreement instead.
- Every Exclude condition met: Exclude.
- The polarized pattern, when switched on: Polarized.
- Otherwise: Uncertain.
Percentages are compared as whole counts, so 70% of 10 needs 7 ratings. Quartiles use the inclusive (type 7) method. The results table shows the median and the interquartile range. Consensus analytics says what else is stored and what that table does not show.
Multi-dimensional rating
Rating & consensus lists the dimensions panelists rate each statement on. A new study has one, Importance, as the primary dimension. Up to five can be set, for example clarity, feasibility, or relevance alongside importance. Only the primary dimension decides consensus.
Controlled feedback (Round 2+)
From Round 2, panelists can see the group's result from the previous round before they rate again. The study's Participant feedback settings choose what is shown. The items below can each be switched off. On a new study, those items start on.
The live panelist block can show:
- The group median
- The middle half of the ratings, labeled "Middle half of ratings" (the first quartile to the third). It does not print the label IQR.
- The share of ratings in each zone
- The panelist's own earlier rating. Other panelists' ratings are never shown.
- Comments, only when the team chose to share them. If no choice is recorded, no comments are shown.
The block's introduction defines the word Status. That sentence is not a Status row, and the block does not show the classification. A statement answered by fewer people than the smallest reportable group shows no figures. A statement the team has reworded since that earlier round withholds the group figures, the person's earlier rating, and the comments. The feedback is built from the previous round's verified results. It is not pinned when the round opens. If a newer results version is computed, panelists see the current one, and the page says so.
Item revision between rounds
Revise is a decision to change the wording. It is not the automatic result of an uncertain or polarized statement.
While another round follows, a statement that is not waiting for a review gets one disposition:
- Advance: rated again as written.
- Revise: reworded in the Item bank, then rated again.
- Hold: not rated again. The statement is settled in or out, or retired.
Those three names are for a round that has another round after it. They are not the label on the last planned round. On that last round, a statement that would have been rated again, and that did not come from a clarity review, is in Final: no consensus. A clarity review that kept the wording or chose a reword, with no round left, is Final: deferred, still unclear. The shorter label No consensus is not either group. An override line uses it when the decision is to rate again and no next round exists, and the protocol rule set uses it as a classification badge. Final: no consensus is not the only last-round result.
While another round remains, the rules send an uncertain, polarized, group-disagreement, or too-few statement on as written. The disposition is Advance. The plan tile for that outcome reads Rated again. Interpreting results maps Uncertain to Advance for that earlier round.
A revised statement keeps its permanent ID, shown as "S12 v2" where the wording matters. The revision cites the round results it responds to. Delphi Studio records the results version, the wording that was rated, and that statement's figures. The team can add a note. Panelists see the current wording. They do not see the earlier round's figures for a statement that was reworded.
The diagram applies only when another round follows. Revise is the team's rewording decision. An uncertain result does not take that path by itself.
After the last round, each statement gets a final result: Final: retained, Final: excluded, Final: retired, Final: no consensus, or Final: deferred, still unclear.
Polarization
A panel split between the two extremes can produce a median in the middle and hide a real disagreement. With the new-study seeds, a statement that is neither retained nor excluded is classified Polarized when at least 30% of ratings fall in each extreme zone, at most 20% fall in the middle, and at least 6 were given. The pattern can reflect ambiguous wording, different interpretations, or distinct groups. Polarized is not Revise. The rule can be switched off, or its numbers changed, before launch. This page does not say whether a study should keep the seeds.
Worked consensus example
Fifteen panelists rate one statement on the 1–9 scale. Another round is planned after this one, so this is not a last round. The check uses the new-study seeds: at least 70% of numeric ratings in the agree zone, and an interquartile range of at most 2. Those seeds are not a universal standard.
Round 1 ratings: 3, 4, 5, 5, 6, 6, 7, 7, 7, 7, 8, 8, 8, 9, 9
| Measure | Value | Retain seed | Met? |
|---|---|---|---|
| Rated 7–9 | 9 of 15 (60%) | at least 70% | No |
| IQR (Q1 5.5, Q3 8) | 2.5 | at most 2 | No |
| Median | 7 | not used | — |
| Result | Uncertain |
Only 1 of 15 rated it 1–3, so it is not excluded, and the disagree share is too small for the polarized pattern. The result is Uncertain. Because another round is planned, the wording stays as written. The disposition is Advance, and the plan tile reads Rated again. Comments about ambiguous wording do not by themselves make the decision Revise. A reword would be a separate decision, and panelists would not then see these Round 1 figures as feedback.
Had 11 of 15 (73%) rated it 7–9 with an interquartile range of 2.5, the percentage seed would be met and the spread seed would not. The statement would stay Uncertain. If another round remained, it would be rated again as written.
Reporting alignment with ACCORD and CREDES
ACCORD is a reporting checklist for biomedical consensus exercises. It guides the report. It does not prescribe the design, and the software does not make a study compliant. CREDES adds Delphi-specific conduct and reporting guidance developed in palliative care. A modified Delphi report should describe panel composition and recruitment, round conduct, feedback, the study's own consensus definition, item changes, stopping, attrition, and limitations.
| Reporting topic | Delphi Studio evidence |
|---|---|
| Panel | Eligibility criteria, Size & mix groups, and the participation report |
| Rounds and feedback | The round plan, the feedback settings, and each round's recorded comment-sharing choice |
| Consensus definition | The rules the study confirmed. What launch freezes is in Launch and amendments. Each results version records the rules it used. |
| Item changes | Permanent statement IDs, and the results each revision cites |
| Attrition | Per-round counts and response rates in the participation report and its CSV |
| Reproducibility | The responses, consensus, and participation CSVs, the decision log, and the study audit CSV |
These records let you take the study's thresholds, zones, and decisions from the record. You still check every manuscript sentence and explain the design. The files are not a finished paper. See Reporting and the ACCORD and CREDES crosswalk.
What the software calculates
- Medians, quartiles, interquartile ranges, and zone percentages for every statement
- The study's confirmed rules, applied to every statement in the round
- Group feedback from the stored results, within the study's settings and the reporting floor
- Each statement's ID and wording history across revisions
- CSV files and reports that name statements by their permanent IDs
It does not compute a mean, a standard deviation, a stability result, or a subgroup comparison.
Next steps
| Topic | Description |
|---|---|
| Rounds | Approve, open, collect, compute results, decide, and prepare the next round. |
| Consensus analytics | Measures, classification, and what results show. |
| Multi-round worked example | A multi-round study with sample data. |
| Round design | The item bank, Rating & consensus, the round plan, and participant feedback. |
| Reporting | Reports and data files for manuscript preparation. |
| Design a defensible modified Delphi study | An explanation of fit, protocol, panel, rounds, stopping, and failure modes. Not a click-by-click setup tutorial. |
Related topics