Skip to main content

Panel Composition Is a Sampling Decision, Not a Recruitment Task

· 4 min read
Delphi Studio Team
Building infrastructure for defensible consensus research

Delphi panels are usually purposive constructions rather than probability samples—which makes their justification more important, not less.

Distinct stakeholder strata deliberately represented in one balanced panel

Conventional power calculations are usually not the basis for Delphi panel size because most Delphi analyses are descriptive rather than inferential. What replaces them is an explicit account of where the relevant expertise sits, how perspectives were partitioned, how candidates were identified, and why the final panel was adequate for the study's purpose. ACCORD accordingly asks authors to report the eligibility criteria, how the panel size was reached, and who selected panel members. See the guideline's explanation and elaboration.

Define expertise operationally​

“Expert” is not a self-explanatory category. Translate it into criteria that fit the research question and can be applied consistently: publication or practice experience in the domain, years in a defined role, institutional responsibility for the process under study, credentialed specialty status, or lived experience when the topic warrants it.

State the criteria in advance and describe the candidate sources—professional society rosters, relevant guideline groups, authorship searches, community organizations, or snowball nomination with a defined process. The test is whether readers can understand why these people, rather than a convenient group, were invited.

Stratify when perspectives may differ predictably​

If academic and community clinicians may rate differently—or clinicians, informaticists, and patients—plan for those perspectives before recruitment. Stratification can prevent one group from numerically overwhelming another and can support transparent within-group or cross-stakeholder analysis.

Stratification also creates obligations. Each stratum needs a recruitment target, monitoring across rounds, and a pre-specified response to small numbers. Small cells can be analytically unstable and may create re-identification risk when feedback or results are broken out by group.

Size the strata, then justify the total​

There is no generally accepted ideal panel size. A recent meta-research study of 1,904 biomedical Delphi studies found wide variation in both planned and retained samples and concluded that clearer guidance is still needed. See the study of sample-size and consensus-method heterogeneity.

That makes a universal rule such as “at least seven per stratum” hard to defend. Instead, justify each stratum against the diversity of expertise required, the precision of percentage-based rules, the feasibility of recruitment, and expected attrition. A 70% threshold in a four-person stratum can only move in 25-point increments; the same threshold in a ten-person stratum moves in 10-point increments. Report counts alongside percentages so readers can see that granularity.

Plan for attrition before it happens​

Attrition is common across repeated rounds, but its rate varies by topic, panel, burden, and design. Decide in advance:

  • Who enters each denominator. Round-specific completers are common, but the denominator and treatment of partial responses must be stated.
  • Whether late entrants or replacements are permitted. A new panelist who did not complete earlier rounds may not have the same exposure to iteration and feedback. If replacements are allowed, define the conditions and report them separately.
  • What happens when a stratum falls below its planned floor. Options include flagging the analysis, suppressing a small-cell breakout, recruiting only if the protocol permits it, or reporting the perspective descriptively rather than making a subgroup consensus claim.

Report invitations, participation, and attrition by round and by relevant stratum. Differential dropout can alter the composition of the panel even when the aggregate response rate appears acceptable.

What the platform tracks​

Delphi Studio stores a primary stakeholder role for each panelist, compares enrollment with recruitment targets, and surfaces balance warnings when groups are missing or dominant. Per-round progress makes attrition visible while a round is open, and optional cross-stakeholder rules can prevent panel-wide agreement from concealing substantial opposition in a sufficiently sized pre-specified group. See Panel management and cross-stakeholder consensus.

Next post: what to do between rounds—feedback design, item revision, and the discipline of not rewriting your way to consensus.