Abstract
Background
Interviews are an integral component of the residency selection process. Many programs
use current residents as interviewers in addition to faculty. Although the reliability
of interview scores between faculty members has been examined, little is known about
the reliability between resident and faculty interviewers.
Objective
This study evaluates the reliability of residents as interviewers compared with faculty.
Methods
A retrospective analysis of interview scores from the 2020–2021 application cycle
was performed at an emergency medicine (EM) residency program. Each applicant participated
in five separate one-on-one interviews led by four faculty members and one senior
resident. Interviewers assigned applicants a score from 0 to 10. Consistency between
interviewers was measured using the intraclass correlation coefficient (ICC). Generalizability
theory was used to measure variance components including applicant, interviewer, and
rater type (resident vs. faculty) and their impact on scoring.
Results
There were 250 applicants interviewed by 16 faculty members and 7 senior residents
during the application cycle. The mean (SD) interview score given by resident interviewers
was 7.10 (1.53) and the mean (SD) score given by faculty was 7.07 (1.69). There was
no statistically significant difference between the pooled scores (p = 0.97). Reliability between interviewers was good to excellent (ICC = 0.90; 95%
CI 0.88–0.92). The generalizability study showed most score variance was attributable
to applicant characteristics and only 0.6% was attributable to interviewer or rater
type (resident vs. faculty).
Conclusions
There was strong concordance between faculty and resident interview scores indicating
reliability of EM resident scoring compared to faculty.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Emergency MedicineAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
REFERENCES
- Program visits and residency interviews.J Emerg Med. 2019; 57: e133-e139
- Emergency medicine residency selection: factors influencing candidate decisions.Acad Emerg Med. 2005; 12: 559-561
Results of the 2018 NRMP program director survey. National Resident Matching Program (NRMP). Accessed December 2, 2020. https://www.nrmp.org/wp-content/uploads/2018/07/NRMP-2018-Program-DirectorSurvey-for-WWW.pdf
- How we select our residents–a survey of selection criteria in general surgery residents.J Surg Educ. 2011; 68: 67-72
- Use of the interview in resident candidate selection: a review of the literature.J Grad Med Educ. 2015; 7: 539-548
- Residency program directors' interview methods and satisfaction with resident selection across multiple specialties.J Am Osteopath Assoc. 2017; 117: 226-232
- Interview techniques utilized in radiology resident selection-a survey of the APDR.Acad Radiol. 2019; 26: 989-998
- Medical school preadmission interviews: are structured interviews more reliable than unstructured interviews?.Teach Learn Med. 2010; 22: 241-245
- Improving methods of resident selection.Laryngoscope. 2010; 120: 2391-2398
- Information collected during the residency match process does not predict clinical performance.Arch Pediatr Adolesc Med. 2000; 154: 256-260
- Residency selection: should interviewers be given applicants’ board scores?.Am J Obstet Gynecol. 2001; 184: 508-513
- Comparison of two methods for ranking applicants for residency.J Am Coll Radiol. 2010; 7: 961-966
- Family practice resident selection: value of the interview.J Fam Pract. 1976; 3: 175-177
- Importance of the faculty interview during the resident application process.J Surg Educ. 2007; 64: 378-385
- Comparative reliability of structured versus unstructured interviews in the admission process of a residency program.J Grad Med Educ. 2011; 3: 517-523
- General surgery residents can be a reliable resource in the evaluation of residency applications.J Surg Educ. 2015; 72: e172-e176
- The economic burden of residency interviews on applicants.Iowa Orthop J. 2018; 38: 9-15
- How much are we spending on resident selection?.J Surg Educ. 2018; 75: e85-e90
- Resident recruitment costs: a national survey of internal medicine program directors.Am J Med. 2013; 126: 646-653
- The role of blinded interviews in the assessment of surgical residency candidates.Am J Surg. 2001; 182: 143-146
- Residency interview video conferencing.Ophthalmology. 2012; 119 (426.e5)
- Virtual multiple mini-interview during the COVID-19 pandemic.Med Educ. 2020; 54: 764-765
- Efficacy of videoconference interviews in the pediatric surgery match.J Surg Educ. 2019; 76: 420-426
Article info
Publication history
Published online: January 31, 2023
Accepted:
January 6,
2023
Received in revised form:
December 17,
2022
Received:
September 29,
2022
Publication stage
In Press Uncorrected ProofFootnotes
Preliminary data included in this article were presented at The Society for Academic Emergency Medicine's national meeting (conducted virtually) in May 2021.
Identification
Copyright
© 2023 Elsevier Inc. All rights reserved.
