Panelist experience
Panelists interact with Delphi Studio through secure magic links rather than traditional accounts or passwords. The interface is intentionally designed to be simple, mobile-friendly, and low-friction, allowing busy clinicians, educators, and other experts to participate between clinical duties, teaching, or other professional commitments.
Access flow
Panelists follow a straightforward path from invitation to rating:
- They receive an email invitation containing a magic link.
- They click the link and are taken directly into the study.
- If consent is required (first visit or after a protocol change), they review and accept the consent form.
- They enter the rating interface for any currently open round.
- They submit ratings and comments. Responses are saved on submit and can be revised until the round closes.
- After completing their ratings, they see a confirmation screen.
If the magic link is expired or invalid, panelists are shown a clear message and can request a new link from the researcher.
Magic links
Magic links provide secure, account-free access to the study.
| Property | Detail |
|---|---|
| Format | /p/[token] — a signed, cryptographically secure URL |
| Scope | Limited to one study and to a single purpose (consent, rate, re-acknowledge, or withdraw); rating links are bound to one specific round |
| Expiry | Emailed links expire 7 days after they are sent (14 days for withdrawal links). The time-to-live on the Consent & comms screen governs links researchers copy from the Panel view |
| Re-send | Researchers can resend links from the Panel view at any time |
Tokens stop resolving once a panelist withdraws. Ending a study closes its rounds, which leaves rating links with nothing open to rate.
Consent screen
On their first visit (or after a protocol amendment), panelists see a consent screen that includes:
- The full consent text configured for the study
- The current protocol version number
- Clear options to accept or decline
Accepting consent moves the panelist to an "accepted" state, granting access to the rating interface, and pins the protocol hash they consented to. Declining routes to the withdrawal flow, where the panelist chooses a data disposition and is recorded as withdrawn. Re-consent uses the same flow but displays the updated protocol information and records the new consent event in the audit log.
Rating interface
The rating screen (/p/[token]/rate) presents the active items for the current open round in a clean, mobile-optimized layout.
| Element | Behavior |
|---|---|
| Item text | Displayed clearly in a single scrolling list, preceded by the round's instructions |
| Scale | Large, tap-friendly inputs for the study's scale: 5- or 7-point Likert, RAND/UCLA 9-point, binary include/exclude, or custom. Studies with multiple rating dimensions show one input per dimension |
| Unable to rate | Valid response when an item falls outside the panelist's expertise; excluded from consensus numerators/denominators and reported separately |
| Comment | Optional by default. When the study's comment rule is set, the field is labelled as required and submission is rejected until a comment accompanies every low rating (at or below the configured threshold), every rated item, or every "Unable to rate" response, depending on the rule |
| Progress | Saved when the panelist submits; they can return through the same link and revise any response until the round closes |
| Statement proposals | When the method allows it, panelists can propose new statements; proposals are de-identified and adjudicated by the research team |
The interface is intentionally minimal to reduce cognitive load for busy professionals.
Round 2+ controlled feedback
In Round 2 or later, panelists see structured feedback drawn from the previous analyzed round before they begin re-rating. Researchers can lock feedback packets before the round opens so every panelist's packet is frozen at the same instant; otherwise each packet is rendered and frozen the first time that panelist views it.
They are shown:
- Group median
- Interquartile range (IQR) and visual spread
- Response histogram
- Zone distribution bar (percentage in Agree / Uncertain / Disagree)
- Their own prior rating for self-comparison
What they do not see:
- Individual ratings from other panelists
- Identities of other panelists
- Researcher-only classifications or dispositions
Comments only appear in feedback after they pass de-identification and the study’s comment approval mode (see Qualitative analysis). Modes range from reviewing every comment to auto-approving comments the redaction pass found clean.
The feedback view becomes immutable after the panelist first views it. This snapshot is stored as a feedbackSnapshot for audit and reproducibility purposes.
Statement proposals
When new item suggestions are enabled for the study, the rating screen accepts one proposed statement per panelist per round, revisable until the round closes. Proposals:
- Are redacted on submission and stored de-identified from other panelists
- Do not enter the item bank until a researcher approves them in Plan next round
- Carry panel provenance when approved so methods sections can report how new items entered the study
Withdrawal
Panelists can withdraw at any point: by declining on the consent screen, or through the withdrawal link carried in the footer of every study email. Replying to a study email to ask for removal is also honored.
Withdrawal asks the panelist to choose what happens to their data:
| Choice | Effect |
|---|---|
| Retain anonymized (default) | Ratings and comments stay in the analysis under the panelist's pseudonym; already-published medians and IQRs remain reproducible |
| Delete | The row linking the pseudonym to the panelist's name and email is purged immediately. Ratings remain, keyed only by the pseudonym |
When a panelist withdraws:
- The action is immediately recorded in the audit log with the chosen data disposition
- They are excluded from all future rounds and their scheduled reminders stop
- They receive a confirmation email, unless they chose deletion — in that case no address remains on file to write to
Withdrawal is always respected and requires no justification from the panelist. Researchers who need to remove a panelist's ratings from an already-analyzed round create a derived re-analysis rather than editing frozen results.
Mobile considerations
The panelist experience is optimized for mobile use:
| Feature | Benefit |
|---|---|
| Large tap targets | Easy rating on phones between meetings or clinical duties |
| No login required | Removes friction for busy experts |
| Revisable submissions | Panelists can return through the same link and change any response until the round closes |
| Quiet hours | Scheduled reminders are shifted out of overnight hours, using the panelist's timezone when one is on record and UTC otherwise |
These design choices help maximize response rates from time-constrained professionals.
What panelists do not see
To protect anonymity and reduce bias, panelists are deliberately shielded from certain information:
| Hidden information | Reason |
|---|---|
| Other panelists' names | Maintains anonymity |
| Individual peer ratings | Prevents anchoring or social influence |
| Consensus classifications | Avoids bias in later rounds |
| Item dispositions | Prevents premature conclusions |
| Study analytics and results | Kept researcher-only until the study concludes |
This controlled information flow is a core strength of the Delphi method.
Researcher responsibilities
Researchers play an important role in creating a positive and effective experience for panelists:
| Researcher task | Impact on panelists |
|---|---|
| Clear and unambiguous item wording | Reduces "Unable to rate" responses and attrition |
| Reasonable round windows (typically 7–14 days) | Respects clinical schedules and improves completion rates |
| Locking feedback packets before opening a round | Ensures every panelist receives fair, identical information |
| Respecting configured quiet hours | Improves response rates and panelist goodwill |
| Prompt follow-up on re-consent or technical issues | Maintains trust and engagement |
Thoughtful configuration by the research team directly contributes to higher-quality data and better retention.
Next steps
- Panel management — recruit and manage experts
- Running rounds — open, close, and iterate
- Qualitative analysis — how comments become feedback