Coding and adjudication
After collection closes, two assigned coders code each comment independently, and an adjudicator settles a comment only where those codes disagree.
Who can open this: Principal investigator, Steering committee (full access), Coder, Adjudicator
Who this is for
Coder is the seat that codes. Saving codes as anyone else returns "Only a coder can code comments." A person who holds Coder and has not been given a slot is told "Assign this person as Coder 1 or Coder 2 on Study team before coding." A further coder is told "This study has two coders already. A third coder cannot code."
The person assigned to "Settles disagreements" settles them, and that person holds Adjudicator. The principal investigator settles one only when they also hold Adjudicator and are the person assigned. Anyone without that settle role is told "Only an adjudicator can settle a disagreement." A principal investigator or an adjudicator who is not the person assigned is told "Only the person assigned to settle coding disagreements on Study team may adjudicate." The person who coded the comment is told "You coded this comment, so someone else has to settle it."
On Study team, assign "Coder 1 (human)", "Coder 2 (human)", and "Settles disagreements". An empty slot reads "Not assigned". The card says "Coders work independently and never see each other's codes. Their names are visible to study leads only, not to panelists or to each other. Assign both human coder slots before coding. Only the assigned adjudicator settles disagreements. A coder slot with history cannot be reassigned; earlier coding remains attributed to its recorded author." The team check says "Assign two people as Coder 1 and Coder 2 before coding." The same person in both coder slots is refused with "Coder 1 and Coder 2 must be different people." Putting a coder in "Settles disagreements" is refused with "The person who settles disagreements cannot also be a coder."
Steering committee (full access) can open the comments page and can see "Agreement between the coders", and cannot save codes unless they also hold Coder. Coordinator, Biostatistics, and Steering confirmer do not open the stage. The round page says "Review is not part of your role" and "You are on this study's team, but your role does not open this stage." That title stays "Review" even on a rating round, whose tab label is "Comments".
While the round is still collecting, the page says "This round is collecting. Comments are coded once collection closes, so what is coded cannot change underneath the coders." A save is refused with "Round {n} is still collecting, so its comments are not final and cannot be coded yet." A finished round says "Round {n} is finished, so its coding can no longer change."
Steps
- On Study team, assign two different people to "Coder 1 (human)" and "Coder 2 (human)", and assign "Settles disagreements". If the comments page can see the counts and fewer than two people hold Coder, it says "Two coders are needed, and this study has {n}." and links "Add the second coder on Study team".
- Close collection on the round. Coding happens on the round's Comments tab (Review, on a questionnaire round). Before the round has opened, the page says "This round has not opened yet. Its comments can be coded once collection has closed."
- A questionnaire round heads the page "Coder comments". A rating round heads it "Comments". A coder sees "Read each comment and tick the ideas in it. You code on your own: you cannot see the other coder's codes, and panelists appear by number only." Someone settling sees "Two coders tag the ideas in every comment without seeing each other's codes. Where they differ, an adjudicator settles it. Panelists appear by number only."
- Filter with "Show". A coder's list includes "Not coded yet". An adjudicator's list includes "Needs adjudication" and the section "Disagreements and comments". "Needs a second look" is available to both. "Question" appears when the round has more than one question.
- Tick the codes and use "Save codes". After the first save the button is "Change codes". A comment already settled refuses with "This comment was settled by adjudication, so its codes can no longer change."
- Where the two coders disagree, open "Settle it", fill "Reason", and use "Settle the disagreement". The hint says the reason is at least 10 characters and at most the field maximum, and "It is kept with the decision." A short reason is refused with "Say why, in at least 10 characters. The reason is kept with the decision." Success reads "Disagreement settled." Settling again is refused with "This disagreement has already been settled." A comment the coders do not disagree on is refused with "The coders do not disagree about this comment."
- People allowed to see agreement (the principal investigator, Steering committee (full access), and Adjudicator) get "Agreement between the coders". It stays empty until both coders have coded a comment: "Agreement appears once both coders have coded a comment." The page then shows "Comments both coded", "Same codes on both", and "Kappa", and "Kappa is worked out over every comment-and-code decision, from each coder's latest codes. It is reported and never stops anything by itself." When Method is comparing kappa, the line is "Meets the study's minimum of {n}." or "Below the study's minimum of {n}. Revise the codebook and recode." When the decision rules are off, it says "The study has no minimum agreement set, so nothing is checked against."
- Final codes raised by enough different panelists are listed under "Structural codes raised by enough panelists". The list uses three panelists. The Thresholds row "Missing structure: independent panelists naming the same gap" is "Not applied yet" while the structural rules are on, with the note "Set and described on Method; no code applies it to responses yet." That row is not what the list counts. On the structural gate those same codes are labeled "Structural codes raised by three or more panelists".
Figure: two coders work apart, and a disagreement is the only place an adjudicator writes a code.
Table: what each comment's status means on the coding page.
| Status | What it means |
|---|---|
| Not coded | Neither coder has saved codes. |
| Waiting for the second coder | One coder has saved codes. |
| Agreed | Both coders saved the same codes. |
| Needs adjudication | The two coders differ. The assigned person can settle it. |
| Settled by adjudication | The disagreement was settled. The codes can no longer change. |
What the app will not do
Comments stay uncoded until collection has closed, and a finished round no longer accepts coding changes. A coder never receives the other coder's codes. Kappa is reported and does not stop a round or change a result. A third person cannot code once both slots are filled. The principal investigator does not code unless they also hold Coder. The codebook is managed by the seats that settle disagreements (the principal investigator and Adjudicator), not by a coder acting only as Coder.
Related
Checked against the app on 2026-10-31.