Interviews are an integral component of the residency selection process. Many programs use current residents as interviewers in addition to faculty. Although the reliability of interview scores between faculty members has been examined, little is known about the reliability between resident and faculty interviewers.


      This study evaluates the reliability of residents as interviewers compared with faculty.


      A retrospective analysis of interview scores from the 2020–2021 application cycle was performed at an emergency medicine (EM) residency program. Each applicant participated in five separate one-on-one interviews led by four faculty members and one senior resident. Interviewers assigned applicants a score from 0 to 10. Consistency between interviewers was measured using the intraclass correlation coefficient (ICC). Generalizability theory was used to measure variance components including applicant, interviewer, and rater type (resident vs. faculty) and their impact on scoring.


      There were 250 applicants interviewed by 16 faculty members and 7 senior residents during the application cycle. The mean (SD) interview score given by resident interviewers was 7.10 (1.53) and the mean (SD) score given by faculty was 7.07 (1.69). There was no statistically significant difference between the pooled scores (p = 0.97). Reliability between interviewers was good to excellent (ICC = 0.90; 95% CI 0.88–0.92). The generalizability study showed most score variance was attributable to applicant characteristics and only 0.6% was attributable to interviewer or rater type (resident vs. faculty).


      There was strong concordance between faculty and resident interview scores indicating reliability of EM resident scoring compared to faculty.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Emergency Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Klammer RM
        • Haydel MJ
        • Gallahue F
        • et al.
        Program visits and residency interviews.
        J Emerg Med. 2019; 57: e133-e139
        • DeSantis M
        • Marco CA.
        Emergency medicine residency selection: factors influencing candidate decisions.
        Acad Emerg Med. 2005; 12: 559-561
      1. Results of the 2018 NRMP program director survey. National Resident Matching Program (NRMP). Accessed December 2, 2020.

        • Makdisi G
        • Takeuchi T
        • Rodriguez J
        • et al.
        How we select our residents–a survey of selection criteria in general surgery residents.
        J Surg Educ. 2011; 68: 67-72
        • Stephenson-Famy A
        • Houmard BS
        • Oberoi S
        • et al.
        Use of the interview in resident candidate selection: a review of the literature.
        J Grad Med Educ. 2015; 7: 539-548
        • VanOrder T
        • Robbins W
        • Zemper E.
        Residency program directors' interview methods and satisfaction with resident selection across multiple specialties.
        J Am Osteopath Assoc. 2017; 117: 226-232
        • Kasales C
        • Peterson C
        • Gagnon E.
        Interview techniques utilized in radiology resident selection-a survey of the APDR.
        Acad Radiol. 2019; 26: 989-998
        • Axelson R
        • Kreiter C
        • Ferguson K
        • et al.
        Medical school preadmission interviews: are structured interviews more reliable than unstructured interviews?.
        Teach Learn Med. 2010; 22: 241-245
        • Prager JD
        • Myer 4th, CM
        • Hayes KM
        • et al.
        Improving methods of resident selection.
        Laryngoscope. 2010; 120: 2391-2398
        • Borowitz SM
        • Saulsbury FT
        • Wilson WG.
        Information collected during the residency match process does not predict clinical performance.
        Arch Pediatr Adolesc Med. 2000; 154: 256-260
        • Smilen SW
        • Funai EF
        • Bianco AT.
        Residency selection: should interviewers be given applicants’ board scores?.
        Am J Obstet Gynecol. 2001; 184: 508-513
        • Collins M
        • Curtis A
        • Artis K
        • et al.
        Comparison of two methods for ranking applicants for residency.
        J Am Coll Radiol. 2010; 7: 961-966
        • Gordon MJ
        • Lincoln JA.
        Family practice resident selection: value of the interview.
        J Fam Pract. 1976; 3: 175-177
        • Brothers TE
        • Wetherholt S.
        Importance of the faculty interview during the resident application process.
        J Surg Educ. 2007; 64: 378-385
        • Blouin D
        • Day AG
        • Pavlov A.
        Comparative reliability of structured versus unstructured interviews in the admission process of a residency program.
        J Grad Med Educ. 2011; 3: 517-523
        • Kim DY
        • Gifford E
        • Nguyen A
        • et al.
        General surgery residents can be a reliable resource in the evaluation of residency applications.
        J Surg Educ. 2015; 72: e172-e176
        • Fogel HA
        • Liskutin TE
        • Wu K
        • et al.
        The economic burden of residency interviews on applicants.
        Iowa Orthop J. 2018; 38: 9-15
        • Gardner AK
        • Smink DS
        • Scott BG
        • et al.
        How much are we spending on resident selection?.
        J Surg Educ. 2018; 75: e85-e90
        • Brummond A
        • Sefcik S
        • Halvorsen AJ
        • et al.
        Resident recruitment costs: a national survey of internal medicine program directors.
        Am J Med. 2013; 126: 646-653
        • Miles WS
        • Shaw V
        • Risucci D.
        The role of blinded interviews in the assessment of surgical residency candidates.
        Am J Surg. 2001; 182: 143-146
        • Pasadhika S
        • Altenbernd T
        • Ober RR
        • et al.
        Residency interview video conferencing.
        Ophthalmology. 2012; 119 (426.e5)
        • Ungtrakul T
        • Lamlertthon W
        • Boonchoo B
        • et al.
        Virtual multiple mini-interview during the COVID-19 pandemic.
        Med Educ. 2020; 54: 764-765
        • Chandler NM
        • Litz CN
        • Chang HL
        • et al.
        Efficacy of videoconference interviews in the pediatric surgery match.
        J Surg Educ. 2019; 76: 420-426