Core concepts

These are the building blocks of a study in Delphi Studio, and the words the app uses for them. The glossary is the word list. The staff guides are the procedures. This page is not a second glossary, and it is not the freeze reference. What launch freezes is only on the governed study record. The measures are only on Consensus analytics.

Study, Setup and launch

A study is one consensus exercise: its panel, its statements, its rules, and its rounds.

You design it in Setup: ten numbered steps in four groups (Study design, Panel design, Round design, Governance), followed by Review & launch. Review & launch is the eleventh screen. It is not one of the ten ready steps. Some steps, such as Method and Rating & consensus, need an explicit confirmation, and a material change withdraws it.

Creating the study makes a draft. A draft is not launched, and creating it does not send an invitation.

The principal investigator launches the study when the draft is ready. Launch creates the planned rounds and moves the study out of draft. It does not send an invitation, and it does not open a round. It also does not freeze every study rule. What launch freezes, what stays editable, and what a recorded amendment can still change are in Launch and amendments. The launch task itself is Review and launch.

Opening a round is later. The principal investigator approves that round, and then the principal investigator or the coordinator opens it. Panelists can answer only after it is open. An invitation is a separate message. It is not launch, and it is not enrollment.

Statement

A statement is one thing the panel rates: a recommendation, competency, indicator, or scenario. Statements live in the Item bank, grouped by area. Each statement has a permanent ID, such as S07, given when it first enters the bank. The ID is never reused and stays the same through every revision and round. A revision shows as "S12 v2", and a revision of a statement that has been rated cites that round's stored results as its evidence.

Round

A round is one cycle of collecting answers. A round is either a questionnaire round, usually Round 1, which gathers the panel's input on structure and wording, or a rating round, in which panelists rate statements on the study's scale.

Round states

State	What it means
Planned	Still only in the round plan. At launch, round 1 is Building and later rounds are Planned. A later round can be prepared once the round before it is in Decision or Closed.
Building	The team prepares the questions and the feedback. The principal investigator approves the round. The approval is tied to the round's content, its rules, the consent version, and the schedule. A relevant change makes it stale.
Open	Panelists answer through their round link. Answers save automatically and can be changed until the cutoff (the deadline plus any grace window).
Reviewing	Collection has closed. Results are computed from submitted responses and reviewed.
Decision	The team records what happens to each statement next.
Closed	A read-only record.

Opening is manual unless Setup sets the round to open on its scheduled date, and even then only after the principal investigator has approved it. Nothing is approved automatically. A late answer is refused at the cutoff.

Rating

A rating is one panelist's answer to one statement in one round. A rating can include:

• A score on each dimension. A new study has one dimension, Importance. A study can add more and names one as primary for classification.
• A comment. Setup decides when a comment is asked for: never, when the rating is low, or when the rating is low or unable to rate.
• Unable to rate, when that option is on. It is not a score. It stays out of every percentage, median, and interquartile range, and the number of panelists who chose it is shown beside the results.

Other panelists never see who gave a rating. Results are computed only after collection closes.

Scale and zones

A study rates on one scale, chosen on Rating & consensus. New studies start from the 1–9 scale. Likert 1–5, Likert 1–7, 0–10, and a custom scale are also offered. Each scale is split into three zones. On the 1–9 scale they are:

Zone	Ratings
Disagree	1–3
Uncertain	4–6
Agree	7–9

Panelists read neutral anchors on the scale. Anchors on a new study do not say Retain or Exclude. Panelists are not shown a Status row or the classification label.

Classification

Consensus here means the rule the study stated in advance for what counts as settled. The software applies that rule. It does not discover that consensus occurred. Agreement, on these pages, means a share of numeric ratings in a zone. Disagreement means the spread or polarization rule the study selected, or the RAND/UCLA disagreement test when that method is selected. Those are different tests. Uncertainty is the status Uncertain: the statement was not retained or excluded, and no other status applied. Polarized, Group disagreement, and Too few responses are different statuses. A narrow spread is not itself consensus. Too few responses is a different class: the statement did not have enough numeric ratings to classify. Appropriateness is the RAND/UCLA reading of a median, withdrawn when that method's disagreement test fires. Concordance is how similarly people ordered the statements. Kendall's W measures concordance. It is not an item-consensus test. The definitions and the formulas stay on Consensus analytics.

After a rating round closes, each statement gets one status from the study's rules. A new study starts from the seeds below. They are editable before launch. They are not a universal standard, and this page does not say whether a study should adopt them.

Status	When, using the new-study seeds
Retain	At least 70% of numeric ratings are in the agree zone, and the interquartile range is at most 2.
Exclude	At least 70% are in the disagree zone, and the interquartile range is at most 2.
Polarized	Neither retained nor excluded, with at least 30% in each extreme zone and at most 20% in the middle, from at least 6 ratings.
Group disagreement	Retained overall, but a stakeholder group of sufficient size opposes it. Applies only when that check is switched on.
Uncertain	None of the above.
Too few responses	Fewer numeric ratings than the study minimum. A new study starts at 7. Nothing else is decided.

These numbers are editable defaults. What launch freezes, and what a recorded amendment can still change, is in Launch and amendments. A median condition can be added, and the spread test can use the disagreement index instead of the interquartile range on a 1–9 scale. That index is not the RAND/UCLA test. A RAND/UCLA study classifies by the panel median and a disagreement test instead, and its statuses read Appropriate, Inappropriate, and Uncertain. The open questions about that test stay on the RAND page. Each study's Thresholds table lists every number it uses and whether it is enforced, reported only, advice only, not applied yet, or off. See Thresholds.

Decision

Between rounds, a person confirms a decision plan. The software does not approve the scientific choice for you.

While another round follows, the disposition names are:

• Advance: rated again as written.
• Revise: reworded in the Item bank, then rated again. Revise is that rewording decision. An uncertain or polarized result is not Revise.
• Hold: not rated again.

A statement still waiting for a review decision has no disposition yet. Interpreting results maps Uncertain to Advance while another round remains. The plan tile for the outcome that sends a statement on unchanged reads Rated again while another round remains. On the last planned round, that outcome is Final: no consensus when the statement did not come from a clarity review. A clarity review that kept the wording, with no round left, is Final: deferred, still unclear. The shorter label No consensus is not either group. An override line uses it when the decision is to rate again and no next round exists, and the protocol rule set uses it as a classification badge. Final: no consensus is not the only last-round result.

After the last round, each statement gets one of five final results: Final: retained, Final: excluded, Final: retired, Final: no consensus, or Final: deferred, still unclear. See After the last round.

Rule-determined outcomes are confirmed by the person the study chose on Study team (the principal investigator or Biostatistics, or Biostatistics only). Outcomes that the rules send to steering review are decided by the steering committee. A decision that departs from what the rules imply is an override and needs a reason of at least 20 characters. The decision log is append-only and hash-chained.

Controlled feedback

Controlled feedback is what panelists see about the group before they rate again. It is not each person's answer, and it is not a live discussion.

The live block can show the group median, the middle half of the ratings (labeled "Middle half of ratings", from the first quartile to the third), the share of ratings in each of the three zones, the person's own earlier rating, and comments the team chose to share. The introduction defines the word Status. That sentence is not a Status row, and the block does not show the classification. It does not print the label IQR, and it does not draw a histogram. A reworded statement withholds the group figures, the earlier rating, and the comments. A reminder that panelists may keep their opinion is shown by default.

Comments are shared only as the study's recorded comment-sharing choice says (verbatim, de-identified, edited, summarized, or withheld). Results for groups smaller than the study's smallest reportable group are withheld. Feedback for Round 2 and later shows the previous round's current verified results. It is not a frozen copy, and the page says so.

Anonymity in the product

Anonymity, as this software implements it, means separation of identity from responses. Response rows do not store a name. Peers do not see peers. After enrollment, ordinary staff screens show a pseudonym such as "ABC-0004".

That is not absolute anonymity. A written reason can reveal an enrolled person's name, and that reveal is audited. Some principal-investigator downloads can contain a name and an email address. The list of those downloads is in Identity and pseudonyms. This page does not close that product question.

Study roles

A person can hold several roles on a study, and their access is the union.

Role	What it is for
Principal investigator	Leads the study and the team. Launches, approves rounds, and completes the study.
Steering committee (full access)	Approves and advises. Decides steering reviews.
Steering confirmer	Decides steering reviews and the structural gate, and nothing else.
Coordinator	Runs recruitment and rounds. Does not launch.
Coder	Codes comments independently.
Adjudicator	Settles coding disagreements.
Biostatistics	Confirms what the rules decided.

Organization administrators have no study authority unless they also hold a study role. The server checks every page and action against these roles. See Roles and access.

How the concepts fit together

The same relationships in words

This list is the same map as the figure.

• Study connects to Setup (method, scale, rules, round plan), to Panel (pseudonyms on ordinary screens), to Item bank (statements with permanent IDs), and to Rounds (opened after approval).
• Setup connects to Launch (creates rounds and does not invite).
• Rounds connect to Ratings and comments.
• Ratings and comments connect to Results versions, sealed after close.
• Results connect to Status from the study rule, and also to Controlled feedback.
• Status connects to Decision plan when another round follows.
• Decision plan connects to Decision log, and on the last round to Five final results.
• Controlled feedback connects back to Rounds.

Next steps

• Glossary: term reference
• The Delphi lifecycle: from Setup to completion
• Structure: how a study is organized
• Modified e-Delphi: the design and the new-study seeds
• Start a new study: create a draft