The Effect of Changing Fellowship Interview Format on Candidate Ranking Variabilities: The COVID-19 Experience

Table 2

Agreement of candidate categories between pairs of interviewers that are within the same room and not within in the same room

Ranking Categories	Interviewer #1 (below)	Interviewer #2			Weighted	95% CI Kappa		KappaP<0.05	Is Kappa in CI of other Kappa?
		Accept	Alternate	Pass	Kappa	Low	High		NWSR
Interviews within the same room	Accept	15 (53.6%)	5 (17.9%)	8 (11.4%)	0.414	0.271	0.557	Yes	No
	Alternate	6 (21.4%)	12 (42.9%)	10 (14.3%)
	Pass	7 (25%)	11 (39.3%)	52 (74.3%)
Interviews not within the same room	Accept	59 (30.7%)	62 (32.3%)	71 (14.8%)	0.159	0.104	0.214	Yes
	Alternate	56 (29.2%)	35 (18.2%)	101 (21%)
	Pass	77 (40.1%)	95 (49.5%)	308 (64.2%)

CI, confidence interval; NSWR, not within the same room; WSR, within the same room. Weighted Kappa outside CI of the other Kappa means there was significant differences in candidate category agreement between WSR and NWS