The Effect of Changing Fellowship Interview Format on Candidate Ranking Variabilities: The COVID-19 Experience
Table 4
Agreement of candidate categories between pairs of interviewers between in-person interviews (2018 and 2019) and virtual interviews (2020).
Ranking Categories
Interviewer #1 (below)
Interviewer #2
Weighted
95% CI Kappa
Kappa
Is Kappa in CI of other Kappa?
Accept
Alternate
Pass
Kappa
Low
High
P<0.05
2019
2018
2018
Accept
17 (22.4%)
18 (23.7%)
41 (18%)
0.086
0.002
0.169
Yes
Alternate
23 (30.3%)
14 (18.4%)
39 (17.1%)
Pass
36 (47.4%)
44 (57.9%)
148 (64.9%)
2019
Accept
24 (31.6%)
17 (22.4%)
35 (15.4%)
0.158
0.072
0.244
Yes
Alternate
20 (26.3%)
15 (19.7%)
41 (18%)
Pass
32 (42.1%)
44 (57.9%)
152 (66.7%)
2020
Accept
20 (26.3%)
25 (32.9%)
31 (16.3%)
0.101
0.015
0.188
Yes
Yes
Yes
Alternate
21 (27.6%)
13 (17.1%)
42 (22.1%)
Pass
35 (46.1%)
38 (50%)
117 (61.6%)
CI, confidence interval. ∗Weighted Kappa within CI of other Kappa means there are no significant differences in candidate category agreement between 2020 and the other years.
