Modified e-Delphi
Modified e-Delphi is the default consensus approach in Delphi Studio. Experts independently rate statements on an ordinal Likert scale, may provide optional or required comments, receive structured and anonymized group feedback, and re-rate until items meet pre-specified consensus criteria or the maximum number of rounds is reached.
Delphi Studio implements this approach with built-in safeguards for methodological rigor: the study wizard captures protocol decisions, the launch snapshot preserves the declared thresholds and rules, analysis pipelines compute statistics consistently, and the item bank maintains version lineage. These features reduce common manual errors and make consequential changes visible while preserving researcher responsibility for item revisions, feedback, stopping, and final classifications.
When to use modified e-Delphi
Use modified e-Delphi when the central research question is whether experts agree that a statement is important, accurate, appropriate for inclusion, or correctly worded. It is the method of choice for developing or validating competency frameworks, quality indicator sets, curriculum outcomes, policy priorities, and similar item banks.
| Use case | Example item |
|---|---|
| Competency frameworks | "A rural telehealth nurse should triage acute chest pain via video within 5 minutes." |
| Quality indicators | "Monthly caseload review is essential for safe community mental health practice." |
| Curriculum outcomes | "Graduates should interpret basic ECG strips without supervision." |
| Policy priorities | "Funding should prioritize preventive care over acute expansion in Year 1." |
Choose modified e-Delphi rather than the RAND/UCLA Appropriateness Method when the task is to assess agreement with normative or definitional statements rather than to rate the appropriateness of specific clinical actions or procedures under defined indications.
A 5-point Likert scale is the platform default because it balances granularity with response burden. A 7-point scale may be selected in the wizard when finer discrimination is desired and panelist fatigue is not a primary concern. See Comparing Delphi types for preset selection guidance.
Default configuration in Delphi Studio
The Modified e-Delphi preset seeds the following values. Review and change them before launch when the protocol requires a different design. Lock and audit policies limit later changes, and all applied choices remain available in snapshots and history.
| Setting | Default | Notes / Delphi Studio implementation |
|---|---|---|
| Anonymity | Anonymous to peers | Identities and individual responses are never visible to other panelists. Researchers use pseudonyms in routine views; authorized identity reveal requires a reason and audit event. |
| Controlled feedback | Enabled | The approved group-level feedback is locked before the round opens; each panelist also receives their own prior response. |
| Item revision | Allowed | Researcher-initiated revisions create new versions linked to parent items; full change history is visible in analytics and exports. |
| New item suggestions | Allowed | Panelists may propose new items in Round 1; these enter the item bank with provenance metadata. |
| Planned rounds | 2 (maximum 3) | Wizard allows a researcher-defined maximum; dashboards track progress toward stopping criteria. |
| Stopping rule | Consensus or max rounds | Study terminates when all items reach consensus or the maximum round is completed; stability metrics are also monitored. |
| Primary dimension | Agreement | Determines consensus classification; additional dimensions appear in reports and exports but do not affect disposition. |
| Scale | 5-point Likert | Anchors are customizable; 7-point option available for studies requiring greater granularity. |
| Agree zone | Ratings 4–5 | Configurable zone boundaries in the wizard. |
| Disagree zone | Ratings 1–2 | Configurable zone boundaries in the wizard. |
| Consensus — agree | ≥80% in agree zone and median ≥ 4 and IQR ≤ 1 | All three conditions must be satisfied simultaneously. |
| Consensus — disagree | ≥80% in disagree zone and median ≤ 2 and IQR ≤ 1 | Symmetric to agree consensus. |
| Bimodality detection | Enabled (20% threshold) | Flags items with ≥20% of ratings in each of the agree and disagree zones, at most 20% in the uncertain zone, and at least 6 ratings; advisory flag for qualitative review that does not change consensus status. |
The rating scale and Likert zones
Panelists rate each item on the chosen Likert scale and may select Unable to rate when they lack sufficient expertise or the item is outside their scope. When the optional rule is enabled, comments are required for low ratings (default threshold ≤ 2) and/or unable-to-rate responses to enrich qualitative analysis.
Unable-to-rate responses are excluded from the denominator when calculating medians, interquartile ranges, and zone percentages. They are, however, tallied separately in round reports so that item relevance and panel expertise coverage can be assessed transparently.
Default 5-point Likert zones
| Score | Anchor (default) | Zone |
|---|---|---|
| 1 | Strongly disagree | Disagree |
| 2 | Disagree | Disagree |
| 3 | Neutral | Uncertain |
| 4 | Agree | Agree |
| 5 | Strongly agree | Agree |
Multi-dimensional rating
In addition to the primary Agreement dimension, researchers may add secondary dimensions on the Rating & consensus step of the wizard. These dimensions provide richer data for analysis and reporting without altering consensus classification, which is always driven by the primary dimension.
| Dimension | Typical use case |
|---|---|
| Importance | Prioritization or ranking studies |
| Clarity | Pre-testing item wording quality |
| Feasibility | Assessing implementation realism |
| Relevance | Scope filtering or item retention |
One dimension is designated primary; the others appear in per-item analytics, round summaries, and all data exports.
Controlled feedback (Round 2+)
Between rounds, the research team reviews and locks a common group-level feedback snapshot before the next round opens. Each panelist receives that same approved group evidence together with their own prior rating. This preserves a reproducible feedback basis without exposing peer-level responses.
Panelists may view:
- Group median (central tendency)
- Interquartile range (IQR) and visual spread indicators
- Histogram of response distribution
- Zone bar showing percentage of ratings in agree / uncertain / disagree zones
- Approved, de-identified comments providing qualitative rationale
- Their own prior rating for self-comparison (peer ratings remain hidden)
Item revision between rounds
Items classified as indeterminate or explicitly flagged for revision during analytics review may be reworded by the research team. Delphi Studio automatically maintains version lineage through parent-item identifiers. Panelists see only the current version in subsequent rounds; researchers and manuscript authors can trace exactly how wording evolved in response to panel feedback.
Revisions are timestamped and linked to their parent items. Where source comments or themes are recorded, those references support a transparent explanation of why wording changed.
Bimodality detection
When bimodality detection is enabled (default), Delphi Studio flags any item in which the share of ratings in each of the agree and disagree zones reaches the study's configured threshold (default 20%), at most 20% of ratings fall in the uncertain zone, and at least 6 ratings are present. Such polarization often signals ambiguous wording, fundamentally different expert interpretations, or distinct subpopulations within the panel. Studies that disable detection get no flag; legacy studies created before the threshold setting existed use a 30% per-tail floor.
The flag is advisory: flagged items receive heightened qualitative review even if median and IQR criteria are technically satisfied, but the flag never changes the computed consensus status by default. The results heatmap surfaces bimodality flags alongside zone classification.
Worked consensus example
Fifteen panelists rate the same item across two rounds.
Round 1 — indeterminate
| Statistic | Value | Threshold | Met? |
|---|---|---|---|
| Median | 3 | ≥ 4 for agree | No |
| IQR | 2 | ≤ 1 | No |
| % rating 4 or 5 | 53% | ≥ 80% | No |
| Classification | Indeterminate | — | Revise and re-rate |
Round 2 — after revision
| Statistic | Value | Threshold | Met? |
|---|---|---|---|
| Median | 4 | ≥ 4 | Yes |
| IQR | 1 | ≤ 1 | Yes |
| % rating 4 or 5 | 87% | ≥ 80% | Yes |
| Classification | Retain (consensus agree) | All three agree rules | Yes |
All three agree-consensus conditions must be satisfied simultaneously: ≥80% in the agree zone, median ≥ 4, and IQR ≤ 1.
If Round 2 had shown median 4 with IQR = 2, the item would remain indeterminate, carry to Round 3 (if rounds remain), or route for researcher review. The platform surfaces these borderline cases in the consensus analytics dashboard with drill-down to individual ratings and comments.
Reporting alignment with ACCORD and CREDES
ACCORD provides the primary reporting checklist for biomedical consensus exercises. CREDES offers complementary Delphi-specific conduct and reporting guidance developed in palliative care. Modified e-Delphi studies should report panel composition and recruitment, round conduct, feedback, consensus definitions, item changes, stopping, attrition, and limitations under the applicable framework.
| Reporting topic | Delphi Studio evidence |
|---|---|
| Panel | Expertise fields, stratum response rates, panel characteristics report |
| Rounds & feedback | Per-round config, locked feedback snapshots |
| Consensus definition | Frozen rule snapshot per round |
| Attrition | Cumulative and per-round response rates, attrition caveats |
| Reproducibility | Launch snapshot, trace bundle |
Delphi Studio exports let authors retrieve the recorded thresholds, zones, and stopping configuration from governed records rather than reconstructing them from memory. Investigators must still verify every manuscript statement and explain the design rationale. See Reporting & exports and the ACCORD and CREDES crosswalk.
Platform advantages
Delphi Studio transforms the modified e-Delphi workflow by:
- Eliminating manual calculation of medians, IQRs, and zone percentages
- Providing consistent, locked group-level feedback while preserving each panelist's own prior response
- Providing real-time dashboards for response rates, item status, and stopping criteria
- Maintaining complete, queryable version lineage for every item revision
- Generating governed evidence and exports that support journal and funder transparency requirements
Next steps
| Topic | Description |
|---|---|
| Running rounds | Open, monitor, lock feedback, and close rounds via the study dashboard. |
| Consensus analytics | Statistics, classification logic, bimodality flags, and drill-down capabilities. |
| Multi-round worked example | End-to-end walkthrough of a multi-round study with sample data. |
| Study wizard — item bank | Importing, editing, versioning, and organizing large item banks. |
| Reporting & exports | Current report, data, and trace artifacts for manuscript preparation. |
| Defensible design tutorial | Methodological fit, protocol, panel, rounds, stopping, and failure modes. |
Related topics