Abstract
Letters of recommendation (LORs) are a central element of an applicant's portfolio for the National Resident Matching Program (known as the “Match”). This is especially true when applying to competitive specialties like emergency medicine (EM). LORs convey an applicant's potential for success, and also highlight an applicant's qualities that cannot always be recognized from a curriculum vitae, test scores, or grades. Traditional LORs, also called narrative LORs, are written in prose and are therefore highly subjective. This led to the establishment of a task force by the Council of Emergency Medicine Residency Directors in 1995 to develop a standardized LOR. Revisions of this form are now referred to as a standardized letter of evaluation. These evaluations in this format have proven to increase inter-rater reliability, decrease interpretation time, and standardize the process used by EM faculty to prepare evaluations for EM applicants. In this article, we will discuss LORs; address applicants' concerns, including from whom to request LORs (EM faculty vs. non-EM faculty vs. non-clinical faculty), number of LORs an applicant should include in his or her application materials, the preferred manner of requesting and the timing in which to ask for an LOR, as well as the philosophy behind waiving the right to see the letter.
Keywords
Introduction
This article discusses common questions regarding letters of recommendation (LORs) for an emergency medicine (EM) applicant—the standardized letter of evaluation (SLOE), formerly known as the standardized LOR (SLOR), how and when to request an SLOE or LOR, the appropriate authors to select, as well as the number of LORs one should attempt to obtain for their application portfolio.
Discussion
How Important Are LORs?
LORs are an integral part of any application for residency. Surveys of EM program directors (PDs) demonstrated that LORs, EM clerkship rotation grades, interview performance, and clinical grades (especially for the core clerkships) are crucial components of the application (
1
, 2
, 3
, 4
, 5
). In fact, LORs from authors in that specialty were the second most commonly cited factor by programs in selecting applicants to interview (4
). Additionally, “global rating” and “competitiveness” on the SLOE were among the highest predictors of top performers in EM residency (6
). In a questionnaire sent to EM PDs in 2013, 93% ranked SLOEs as the most important factor when selecting applicants for interviews (7
). Additionally, a recent study indicated that more weight is placed on SLOEs of applicants from away institutions compared to those from home institutions (5
).What is the Purpose of LORs?
LORs are written to communicate information about an individual to an institution, training program, or selection committee (
8
). LORs communicate an applicant's potential for success. They also describe and emphasize qualities not quantified elsewhere in an application. EM PDs, faculty, programs, and residents, in some cases, look for applicants with skills required to be great emergency physicians. These include getting along well with others, critical and efficient thinking, leadership and team-member qualities, the ability to handle difficult situations, outstanding communication skills, professionalism, humility, and potential. This information is not commonly communicated through one's curriculum vitae (CV), clinical grades, or test scores, but can be conveyed in a recommendation letter relatively easily. Therefore, LORs assume a critical role in candidates' application materials, and thus in the National Resident Matching Program (or the “Match”) (2
).What Is the SLOE?
Traditional LORs, also called narrative LORs (NLORs), are written in prose using highly subjective terminology with poor inter-rater reliability when interpreted (
9
, 10
). Accordingly, the Council of Emergency Medicine Residency Directors (CORD) established a special task force in 1995 to develop the SLOR (currently known as the SLOE), to increase the inter-rater reliability and standardize the process of evaluation of applicants (3
, 11
). The SLOE is a three-page document, used exclusively by EM faculty, consisting of four parts—background information on the author and applicant (including grade information); assessment of the applicant's qualifications for EM (compared to other applicants or peers); global assessment of the applicant (with strict guidelines for authors); and narrative comments (currently limited to 250 words) (2
, 11
). There is also an opportunity for additional narrative comments to summarize important specifics about the author's institution or rotation (also limited to 250 words). The SLOE provides a place to mark whether or not the applicant (“student”) has waived the right to see the letter, and also to date and electronically sign (12
). The SLOE is generally updated annually by CORD; it is available at the CORD EM website (www.cordem.org/) (see Appendix). EM faculty are expected to use the electronic SLOE, which is completed in real time on the Internet through the website. There are three distinct SLOE forms: the first is used by EM faculty who have an established EM residency program at their institution, the second is an alternate SLOE for authors from a sub-specialty rotation, and the third form is another alternate SLOE to be used by practicing emergency physicians without an established EM program at their institution (non-EM Residency Faculty). Each of these links to the respective electronic form encourages authors new to the process to seek mentorship from CORD or program leadership (if applicable). All authors must answer the question at the beginning of each SLOE “I have read this year's instructions at www.cordem.org/” before continuing. These forms differ slightly in the qualifications for EM assessment (section B) and the written comments (section D). There is also a question in section A about which EM rotation generated the candidate's SLOE; there is, however, a lack of agreement as to whether or not mandatory exposure to the emergency department (ED) for a short time early in medical school should be counted as a rotation. There is also a question in section A asking for the dates of the rotation (due to confusion and because a few students may provide inaccurate information to evaluators). According to the CORD website, this instrument (the SLOE) “… remains flexible and reflective of membership opinion” (CORD Standardized Letter of Evaluation (SLOE)
Council of emergency medicine residency directors.
Council of emergency medicine residency directors.
https://www.cordem.org/resources/residency-management/sloe/
Date accessed: September 19, 2018
12
). Non-EM authors write NLORs.CORD Standardized Letter of Evaluation (SLOE)
Council of emergency medicine residency directors.
Council of emergency medicine residency directors.
https://www.cordem.org/resources/residency-management/sloe/
Date accessed: September 19, 2018
Who Should I Select as an Author?
In general, the best authors of LORs are experienced faculty (or nonclinical faculty in unique cases) who know an applicant well in an area related to EM. They can describe specific exemplary behaviors they observed in the narrative portion of their letter (
8
, 13
). Original versions of the SLOR included a global assessment score (GAS), rated by evaluators from good to outstanding, and likelihood of matching assessment (LOMA), rated by evaluators from unlikely match to very competitive (14
). A retrospective study that reviewed SLOEs written for three EM residency program applicants during the 2012 match revealed that more experienced authors were more likely to write SLOEs with a GAS of “Outstanding” and a LOMA of “Very Competitive” compared to less-experienced authors (15
).EM Faculty
When selecting an EM faculty author, it is essential to have extended clinical contact with that individual. Working 12 clinical hours with an attending is a good starting point for considering them as a potential author, because this might be slightly more than one clinical shift and perhaps 8–12 different cases. This is important because it affords the opportunity to witness learning and improvement in professionalism, data gathering, documentation, stress and time management, examination, clinical, and (in some cases) procedural skills (
16
). Comprehensive diagnostic approaches, appropriate treatment plans, overall efficiency, and a strong work ethic witnessed by a potential faculty author should improve the letter's content and accuracy (16
, 17
). The length of time that the SLOR's author knew an applicant was significantly correlated with GAS and LOMA scores given to the applicant (15
).Following completion of each EM rotation, the clerkship or program director generally writes SLOEs for applicants based on personal observation and composite evaluations. These data points may be gathered during clinical shifts in the ED or from classroom activities; journal club participation; simulation cases; faculty meetings; and encounters with staff (including nurses and administrators), other students, residents, or faculty. This evaluation may be requested or offered. Many clerkships offer “departmental,” “consensus,” or “group” SLOEs, which CORD encourages in their instructions to authors “…for institutions that see a moderate number of students” (
12
). Although perhaps more objective, group or departmental SLOEs may not reflect the personal experience that individual faculty had with an applicant (CORD Standardized Letter of Evaluation (SLOE)
Council of emergency medicine residency directors.
Council of emergency medicine residency directors.
https://www.cordem.org/resources/residency-management/sloe/
Date accessed: September 19, 2018
14
). Content for the SLOE is obtained using input from clerkship directors, PDs, or clinical faculty and residents; a balanced perspective of the applicant is provided when written appropriately. It is a good idea for applicants to learn who writes SLOEs (and whether or not they are written as a group SLOE) at the beginning of their EM clerkship, although it is best if students interact positively with as many people as possible. Furthermore, students should inquire about the procedure for trading shifts if necessary, so they can secure at least one or two shifts with key faculty. Changes to the schedule may also be needed if an important commitment prevents a student from making a scheduled shift. Residency applicants need not fear the validity of the content found in group SLOEs. These letters are often preferred by PDs because the content is derived from multiple individuals rather than the opinion of a single faculty member.Whether or not you should ask additional EM faculty for an SLOE generated from a single ED is unclear. A great experience with a senior EM faculty member might warrant requesting a letter from him or her. However, SLOEs from more than one faculty member at the same institution during the same clerkship often provide similar information and may not be useful. If possible, an SLOE from each EM clerkship is recommended because clinical perspectives from EM faculty at distinct clinical sites provide greater information to the selection committee and program leadership than two SLOEs from the same clerkship. The Electronic Residency Application Service (ERAS) allows you to select which SLOEs or NLORs be downloaded for your application materials.
If you find yourself completing an EM clerkship at an institution where faculty are unfamiliar with the SLOE (perhaps a community ED with few students or an ED without affiliation with an EM residency program), it is helpful to give your faculty author a description of the SLOE, the CORD website address, and instructions. Diplomatically stress to the author that the SLOE is an integral part of the application process, should be taken seriously, and needs to be completed in a professional manner (preferably as an online document, not downloaded, handwritten, and then uploaded, for example).
Non-EM Faculty
Although controversial, consider requesting a narrative letter (or in some unusual situations, letters) from non-EM faculty. These are generally from academic faculty with whom you worked closely for an extended period on core rotations, such as internal medicine or surgery. This might also include non-EM faculty with whom you have done research or served as a course assistant for an extended period. Academic faculty on non-EM rotations will have worked with you under different yet difficult conditions, perhaps more longitudinally, and might be able to share details about your performance and personality over an extended period of time. Therefore, while on all of your rotations, make an effort to get to know key teaching faculty who might serve as possible references. Furthermore, a strong Internal Medicine (IM), Family Medicine (FM), or Critical Care faculty NLOR is important for applicants applying to combined EM and IM or EM and FM programs. Similarly, if applying for a combined EM and Pediatrics programs, a strong NLOR from a senior pediatrics faculty at a residency program commenting on your pediatrics clerkship performance in addition to your transcript grade and a paragraph buried in your Medical Student Performance Evaluation (MSPE) might prove advantageous.
Nonclinical Faculty
If you spent significant time volunteering or working in your medical school's office of curricular affairs and developed an important relationship with a leader or administrator during this experience, consider requesting an NLOR from that individual. Evidence of your character, integrity, and potential can be described outside of your performance during a clerkship, so it may be helpful to have someone speak to your abilities and personality outside of the hospital (
18
). This might also hold true if you spent a considerable amount of effort and energy with nonclinical faculty performing research or other work during medical school.Does the Academic Rank of My Evaluator Matter?
The ideal author of your evaluation is a faculty member who is known to faculty at EM residency programs, widely respected, and has a history of writing fair and accurate letters (
2
). However, it is more important to have your evaluation written by someone who knows you well and has worked with you closely, as long as this individual is a faculty member and not an EM resident. As is often the case, junior faculty tend to be more “visible; ” therefore, you may have more exposure to them in clinical and nonclinical (teaching) settings during your clerkship. It is unwise to request a letter from a prominent faculty member simply because of their seniority; a nondescript or inaccurate letter from a senior faculty may be unhelpful or even detrimental to your application. The best situation is to obtain an SLOE from an experienced or senior emergency physician who knows you well, is comfortable mentoring medical students, is familiar with the SLOE, has spent time directly observing your clinical activities, and is known by PDs around the country (18
).Junior faculty tend to work many clinical shifts, which makes them readily accessible. They are generally energetic and approachable, and often closer to the application process than senior faculty. While it may be tempting to ask very junior faculty for a letter, they may be unfamiliar with the SLOE, be uncomfortable providing a strong recommendation (fearing their reputation), and have less experience writing SLOEs and estimating potential (
19
). The CORD website SLOE instructions encourage authors new to the process to “…seek mentorship from CORD…or…program leadership before completing the SLOE.” Clerkship and program directors generally have extensive experience reading and writing SLOEs and are, therefore, more likely to compose meaningful letters (19
, 20
).How Many SLOEs or NLORs Do I Need?
There is no definitive answer to the number of SLOEs or NLORs an applicant requires. At least two SLOEs should be written by EM faculty, ideally from different EM clerkship experiences (
19
). SLOEs from multiple faculty at the same institution should be avoided if possible (but not always), as they generally provide similar information. In addition, it is reasonable to consider having one letter from a non-EM clinical or non-clinical faculty (18
). Generally speaking, if you had a great experience with a senior faculty in any setting, especially over an extended period, it is worth considering them as an author of a strong LOR.How Do I Ask for an LORs?
When should I ask?
Ask your author if he or she is willing and able to write a supportive LOE on your behalf at a time when your interactions are recent. Consider asking near the end of your EM clerkship after working several shifts with that person. Because LORs take time to write, give authors as much advanced notice as possible. Ask at least 1 month before the due date to be respectful of the author's clinical, academic, and travel schedules (
18
).Should I ask in person?
It is best to schedule an appointment with your reference author to request an LOR or LOE. This allows your author to learn more about you and your goals, which will help him or her write a superior letter (
18
, 21
). In some cases, a faculty may offer to write a letter during clinical duties. While your instinct may be to accept this offer immediately, it is best to express gratitude yet still plan a meeting outside the ED. If you are unable to meet in person, communication by phone or e-mail is acceptable.What should I say?
When asking a faculty member for a letter, it is important to provide the author with an opportunity to decline, as authors generally only accept a request if they are able to be supportive of an applicant (
19
, 22
). A skillful question to ask is: “Do you feel you have gotten to know me well enough to write a strong letter of evaluation (or support) for a residency position in EM?”If an experienced faculty member offers to write you a letter, he or she was likely impressed with your potential as a resident and future faculty, and would therefore like to advocate on your behalf. Consider accepting this offer, especially if it is from a senior faculty in an important leadership position. It is still important to attempt to meet outside of the ED.
What Materials Should I Provide My Reference Author?
Prepare a concise but relevant information packet for your authors to help them write a strong and accurate letter. Give them the opportunity to know you better by providing a statement of purpose (such as your strengths and goals, and how you plan to achieve them), your personal statement, an updated CV, and any writing samples that have been published (one or two maximum). Clarify your roles in any research, publications, and volunteer activities (
23
). Encourage your authors to look over these materials prior to writing your letter, and make yourself available if they have any questions (20
).Provide your author with any necessary forms and deadlines. For non-EM faculty, provide a description of the type of EM program you are applying to and why. Be sure to give your reference author clear instructions on how to submit this letter. Most if not all NLORs are submitted through the dean's office at the medical school and uploaded directly through the ERAS; it is possible that authors might not be familiar with this process. If for some reason this LOR requires submission by U.S. mail, provide your author with a stamped, correctly addressed envelope (
23
). In other words, demonstrate your attention to detail while making it as easy as possible for your author to complete this task.Should I Waive My Right to See the Letter?
The answer to this question is almost always (and resoundingly) “yes.” If authors know that an applicant will be able to access his or her letter, the content and language of the letter, and what it includes or omits, may change. As a result, letters written for applicants who have not waived their right to see the letter according to the Family Educational Rights and Privacy Act (FERPA) may be biased and raise concern about their accuracy (
10
). The bulk of applicants to 14 EM programs (93%) waived their FERPA rights to see their SLORs or NLORs. In one multicenter study, applicants who retained their FERPA rights were more likely to receive a GAS of “Outstanding” than those who did waive their rights, although this retrospective study had numerous limitations identified by the authors (24
).Conclusions
When considering SLOEs or NLORs for EM residency training, preparation about and insight into the process are essential. The best authors are faculty who know an applicant well, are experienced at composing SLOEs or NLORs, and are able to accurately identify potential and predict future success. Therefore, keep your eyes open for faculty whom you admire and who make an effort to get to know you during your clinical rotations or outside activities, particularly those related to EM. When requesting a letter, ask a potential author shortly after a time when your interactions were memorable. When possible, request a strong reference well in advance of any deadlines, so that the author will feel less pressured (it is likely that authors will have multiple evaluation requests from other students with similar deadlines). If possible, schedule a brief appointment with that individual to request the letter and answer any questions. Bring a concise, prepared portfolio and offer it to the faculty author, in case they wish to refer to it. This does not necessarily need to include grades in other clerkships, and do not be upset if they decline to take it. This “package” should include detailed information about your goals and accomplishments so authors can write an ideal LOE about you that supports your efforts to achieve your goals.
Acknowledgments
This work was supported by the Department of Emergency Medicine at the American University of Beirut Medical Center. The funding source was not involved in the preparation of the article, writing of the report, or decision to submit the article for publication. This research received a non-restricted educational grant from the not-for-profit Resident Student Association (American Academy of Emergency Medicine) and the Department of Emergency Medicine at the American University of Beirut.
Supplementary Data
- Appendix 1
References
- Characteristics of the emergency medicine standardized letter of recommendation.Acad Emerg Med. 2000; 7: 409-410
- Letters of recommendation.in: Rogers R.L. Moayedi S. Medical student educators’ handbook. Clerkship Directors in Emergency Medicine, Lansing, MI2010: 285-298
- A comparison of standardized and narrative letters of recommendation.Acad Emerg Med. 1998; 5: 1101-1104
- Results and data: 2018 main residency match®.National Resident Matching Program, Washington, DC2018
- Emergency medicine residency selection criteria: an update and comparison.AEM Educ Train. 2018; 2: 146-153
- Predictors of a top performer during emergency medicine residency.J Emerg Med. 2015; 49: 505-512
- Council of emergency medicine residency directors' standardized letter of recommendation: the program director's perspective.Acad Emerg Med. 2014; 21: 680-687
- Ethics seminars: beyond authorship requirements—ethical considerations in writing letters of recommendation.Acad Emerg Med. 2001; 8: 70-73
- Reliability in evaluating letters of recommendation.Acad Med. 2000; 75: 1029
- Improving the letter of recommendation for pediatric residency applicants: an idea whose time has come?.J Pediatr. 2003; 143: 143-144
- A standardized letter of recommendation for residency application.Acad Emerg Med. 1999; 6: 1141-1146
- Council of emergency medicine residency directors.(Available at:)https://www.cordem.org/resources/residency-management/sloe/Date accessed: September 19, 2018
- Letters of recommendation: rating, writing, and reading by clerkship directors of internal medicine.Teach Learn Med. 2009; 21: 153-158
- Analysis of the evaluative components on the Standard Letter of Recommendation (SLOR) in emergency medicine.West J Emerg Med. 2014; 15: 419
- Does the experience of the writer affect the evaluative components on the standardized letter of recommendation in emergency medicine?.J Emerg Med. 2014; 46: 544-550
- Emergency medicine standardized letter of recommendation: predictors of guaranteed match.Acad Emerg Med. 2001; 8: 648-653
- The outstanding medical student in emergency medicine.Acad Emerg Med. 2001; 8: 402-403
- Letters of recommendation.in: Kazzi A.A. Schofer J.M. AAEM’s rules of the road for medical students: the guide for a career in emergency medicine. American Academy of Emergency Medicine, Milwaukee, WI2003: 65-72
- Mentoring medical students in academic emergency medicine.Acad Emerg Med. 2004; 11: 1351-1357
- Inaccuracy of the global assessment score in the emergency medicine standard letter of recommendation.Acad Emerg Med. 2010; 17: S38-S41
- Gender and the council of emergency medicine residency directors standardized letter of recommendation.Acad Emerg Med. 2004; 11: 988-991
- Letters of recommendation: what does ‘good’ really mean?.Acad Emerg Med. 1997; 4: 833-834
- An educator’s “scoop” on asking for or writing a recommendation.MLO Med Lab Obs. 2009; 41: 26-28
- A multicenter study of the Family Educational Rights and Privacy Act and the standardized letter of recommendation: impact on emergency medicine residency applicant and faculty behaviors.J Grad Med Educ. 2014; 6: 292-295
Article info
Publication history
Published online: July 30, 2019
Accepted:
April 9,
2019
Received:
March 26,
2019
Identification
Copyright
© 2019 The Authors. Published by Elsevier Inc.
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0)
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy
