Qualitative analysis

Open-ended comments in Delphi studies often contain valuable explanatory information — why experts agreed or disagreed, what wording was unclear, and what evidence or experience shaped their views. Delphi Studio provides a governed, AI-assisted workspace where the system proposes themes and researchers retain full control through structured review and approval.

This approach combines the efficiency of AI with the judgment and accountability of human researchers.

Two surfaces for qualitative work

Delphi Studio offers two related but distinct workspaces for qualitative analysis:

Surface	Scope	Best used for
Round-scoped qual run	One closed round	Systematic coding of comments after each round closes
Study-wide workspace	All rounds across the entire study	Cross-round theme synthesis, codebook development, and final reporting

Use round-scoped runs for consistent, per-round analysis. Use the study-wide workspace when you need to synthesize themes across multiple rounds or prepare final qualitative findings for publication.

The qualitative workspace shows theme provenance, adjudication gates, and export options for the study-wide codebook.

Pipeline overview

When a round closes, comments enter a structured pipeline with three main passes:

1. Pass 1 – Inductive: The system reviews raw (redacted) comments and proposes initial themes.
2. Pass 2 – Applied: Researchers review and refine the emerging codebook, then apply it across comments.
3. Pass 3 – Synthesis: Broader patterns are identified to support item revision decisions and narrative reporting.

Human researchers must approve work at key points before the pipeline continues.

Redaction first

Before any comment is sent to an AI model or included in panelist feedback, it passes through a redaction layer:

• Automated regex patterns remove obvious identifiers such as emails, phone numbers, and common name patterns.
• As defense-in-depth, a roster-aware scrubber additionally removes panelist pseudonyms and roster identifiers from AI outputs before they are stored.
• Future enhancements will include more advanced named-entity recognition.
• Only redacted text is ever sent to language models or exported to REFI-QDA.

Comment approval modes

Before comments appear in controlled feedback, they pass redaction and a study-level comment approval mode (configured on Rounds & feedback in the wizard):

Mode	Behavior	When to use
review_all	Every comment requires researcher approval before feedback (default for new studies)	Maximum governance; CREDES-sensitive panels
auto_when_clean	Comments the redaction pass found clean are auto-approved; flagged comments still need review	Lower friction when volume is high and risk is lower

Comments that are not approved never reach panelists in subsequent rounds.

Adjudication gates

Five human review gates pause the pipeline and require researcher action:

Gate	Researcher decides
Codebook	Accept, edit, merge, or reject AI-proposed themes
Feedback	Which redacted comments should appear in the next round's controlled feedback
Disposition	Which qualitative evidence supports retaining, revising, or dropping specific items
Domain report	Narrative synthesis for a domain-scoped run
Study report	Narrative synthesis and qualitative findings across the whole study

In addition, comments the pipeline cannot code reliably open a needs-human-coding adjudication that routes them to a researcher for manual coding.

Report adjudications can be sent back for AI revision, but revision cycles are capped at 2 — beyond that, the adjudication must be resolved by the research team directly. Each gate can be claimed by a researcher. Claims use a soft 30-minute lock so that other team members can resume work if a gate is left unclaimed.

Theme provenance and codebook

Every theme carries clear provenance so researchers can understand its origin:

Provenance	Meaning
human	Created entirely by a researcher
ai_proposed	Suggested by AI (awaiting or after researcher acceptance)
ai_refined_human	AI draft that was edited or refined by a researcher

The system also records the model ID and prompt version (as a content-addressed hash) for every AI interaction, supporting reproducibility and audit requirements.

The codebook supports:

• Parent/child hierarchy (e.g., broad domains with sub-themes)
• Locking to prevent unintended changes before export
• Audited split and merge operations with full theme lineage
• An agreement metric: the percentage of AI-applied codes confirmed without human override (no kappa statistic is computed)
• Round carry-forward: each round's codebook starts from the prior round's approved codebook, recorded with round_carry provenance

Each AI-proposed theme also carries a language-drift score comparing the theme definition against the source comment vocabulary. Scores below 0.6 surface a non-blocking warning in the UI so researchers can check whether the theme has drifted from what panelists actually wrote.

Starting a qualitative run

To begin qualitative analysis after a round:

1. Confirm the round is closed and all ratings are finalized.
2. Navigate to the round detail page and select Qualitative analysis.
3. Start the run. This enqueues a durable background workflow.
4. Work through the adjudication gates as they appear.
5. Use approved themes to inform item revisions and the content of feedback packets for the next round.

Each run is scoped to either a single domain or the whole study, and is pinned to the study's launch snapshot — later protocol edits cannot change what the run analyzed. The exact version of every pipeline prompt (prompts A–I) is also pinned per run, so once a run starts its prompts and models are fixed.

Budget and cost controls

Because AI usage incurs cost, Delphi Studio includes several controls:

• Budget caps are set at the environment level via configuration variables.
• If a run reaches its budget limit, it enters a paused_budget state; a per-run budget adjustment is available only when resuming a paused run.

Model routing goes through the Vercel AI Gateway with fixed per-task model profiles: the qualitative pipeline prompts (A–I) run on Claude models, while feedback summaries, manuscript drafting, deductive coding, and theme proposals currently run on gpt-5-thinking. A local Llama provider exists for development and offline use only; the earlier per-study provider override has been retired. Sending any study data to an external model requires the study's external AI opt-in (external_ai_allowed), and every external call is recorded in the external_ai_calls audit table.

Exports

Once qualitative work is complete, several export options are available:

Export	Contents
REFI-QDA	Redacted comments with applied codes (importable into NVivo, MAXQDA, ATLAS.ti)
Qual audit ZIP	Themes, code frequencies, revision evidence, and audit details
Manuscript	AI-assisted qualitative results section (requires human review and approval)

Governance principles

Qualitative analysis in Delphi Studio is built around five core principles:

Principle	Implementation
AI proposes, humans dispose	No AI-generated content reaches panelists or final reports without researcher approval
External AI is opt-in	No study data is sent to an external model unless the study explicitly enables external_ai_allowed; every external call is audited in external_ai_calls
De-identification first	Comments are redacted before being sent to any language model or included in feedback
Auditability	Every AI call is logged with model, prompt version, and payload hash
Reproducibility	Prompts and model versions are frozen and recorded for later verification

These principles ensure that qualitative analysis enhances — rather than replaces — researcher judgment while maintaining transparency and methodological integrity.

Next steps

• Reporting & exports — REFI-QDA and qual audit ZIP
• Audit trail — AI call logging
• Running rounds — feedback lock timing