Interviews are an integral component of the residency selection process. Many programs use current residents as interviewers in addition to faculty. Although the reliability of interview scores between faculty members has been examined, little is known about the reliability between resident and faculty interviewers.
This study evaluates the reliability of residents as interviewers compared with faculty.
A retrospective analysis of interview scores from the 2020–2021 application cycle was performed at an emergency medicine (EM) residency program. Each applicant participated in five separate one-on-one interviews led by four faculty members and one senior resident. Interviewers assigned applicants a score from 0 to 10. Consistency between interviewers was measured using the intraclass correlation coefficient (ICC). Generalizability theory was used to measure variance components including applicant, interviewer, and rater type (resident vs. faculty) and their impact on scoring.
There were 250 applicants interviewed by 16 faculty members and 7 senior residents during the application cycle. The mean (SD) interview score given by resident interviewers was 7.10 (1.53) and the mean (SD) score given by faculty was 7.07 (1.69). There was no statistically significant difference between the pooled scores (p = 0.97). Reliability between interviewers was good to excellent (ICC = 0.90; 95% CI 0.88–0.92). The generalizability study showed most score variance was attributable to applicant characteristics and only 0.6% was attributable to interviewer or rater type (resident vs. faculty).
There was strong concordance between faculty and resident interview scores indicating reliability of EM resident scoring compared to faculty.
To read this article in full you will need to make a payment
Purchase one-time access:Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
One-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:Subscribe to Journal of Emergency Medicine
Already a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
- Program visits and residency interviews.J Emerg Med. 2019; 57: e133-e139
- Emergency medicine residency selection: factors influencing candidate decisions.Acad Emerg Med. 2005; 12: 559-561
Results of the 2018 NRMP program director survey. National Resident Matching Program (NRMP). Accessed December 2, 2020. https://www.nrmp.org/wp-content/uploads/2018/07/NRMP-2018-Program-DirectorSurvey-for-WWW.pdf
- How we select our residents–a survey of selection criteria in general surgery residents.J Surg Educ. 2011; 68: 67-72
- Use of the interview in resident candidate selection: a review of the literature.J Grad Med Educ. 2015; 7: 539-548
- Residency program directors' interview methods and satisfaction with resident selection across multiple specialties.J Am Osteopath Assoc. 2017; 117: 226-232
- Interview techniques utilized in radiology resident selection-a survey of the APDR.Acad Radiol. 2019; 26: 989-998
- Medical school preadmission interviews: are structured interviews more reliable than unstructured interviews?.Teach Learn Med. 2010; 22: 241-245
- Improving methods of resident selection.Laryngoscope. 2010; 120: 2391-2398
- Information collected during the residency match process does not predict clinical performance.Arch Pediatr Adolesc Med. 2000; 154: 256-260
- Residency selection: should interviewers be given applicants’ board scores?.Am J Obstet Gynecol. 2001; 184: 508-513
- Comparison of two methods for ranking applicants for residency.J Am Coll Radiol. 2010; 7: 961-966
- Family practice resident selection: value of the interview.J Fam Pract. 1976; 3: 175-177
- Importance of the faculty interview during the resident application process.J Surg Educ. 2007; 64: 378-385
- Comparative reliability of structured versus unstructured interviews in the admission process of a residency program.J Grad Med Educ. 2011; 3: 517-523
- General surgery residents can be a reliable resource in the evaluation of residency applications.J Surg Educ. 2015; 72: e172-e176
- The economic burden of residency interviews on applicants.Iowa Orthop J. 2018; 38: 9-15
- How much are we spending on resident selection?.J Surg Educ. 2018; 75: e85-e90
- Resident recruitment costs: a national survey of internal medicine program directors.Am J Med. 2013; 126: 646-653
- The role of blinded interviews in the assessment of surgical residency candidates.Am J Surg. 2001; 182: 143-146
- Residency interview video conferencing.Ophthalmology. 2012; 119 (426.e5)
- Virtual multiple mini-interview during the COVID-19 pandemic.Med Educ. 2020; 54: 764-765
- Efficacy of videoconference interviews in the pediatric surgery match.J Surg Educ. 2019; 76: 420-426
Published online: January 31, 2023
Accepted: January 6, 2023
Received in revised form: December 17, 2022
Received: September 29, 2022
Publication stageIn Press Uncorrected Proof
Preliminary data included in this article were presented at The Society for Academic Emergency Medicine's national meeting (conducted virtually) in May 2021.
© 2023 Elsevier Inc. All rights reserved.