USMLE Step 1 scores get far more weight than they deserve in the residency selection process, experts say, and that’s harming students. Now for the first time, key stakeholders have come together to reform USMLE — and the entire selection process. Each year, tens of thousands medical students take the United States Medical Licensing Examination (USMLE) knowing that if they don’t pass, they can’t practice medicine. But the pressure extends well beyond that. Scores on the exam — particularly Step 1 of the three-part test — can dramatically affect an applicant’s chances at their desired residency slot. Experts consider that troubling for several reasons, including that the test was never designed for residency selection purposes. What’s more, anxiety over USMLE scores — and other application-related stresses — are taking a tremendous toll on medical students’ well-being. “Students think this is going to determine the trajectory of their careers. That’s terrifying,” says Christle Nwora, a third-year student at McGovern Medical School in Houston, Texas. “As it’s used now, this test is causing us harm.” Certainly, there is basis for such fears: Some residency programs will screen out applicants who don't have a Step 1 score well above the passing mark. Yet without USMLE scores, residency programs have trouble sifting through the piles of applications, which only continue to grow each year. “For three slots, you can get hundreds of applications,” says Kenneth Simons, MD, senior associate dean for graduate medical education at the Medical College of Wisconsin (MCW). “Program directors are just swamped.” Now, for the first time, leaders in academic medicine are uniting to tackle the thorny challenges of the current residency selection process. The Invitational Conference on USMLE Scoring (InCUS) recently brought together more than 60 students, residents, program directors, deans, and members of the public to discuss underlying problems and identify possible solutions. The two-day event was convened by five leading medical groups: the AAMC, the American Medical Association, the Educational Commission for Foreign Medical Graduates, and the two groups that oversee the USMLE, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME). Using brainstorming activities, reams of reading materials, and extensive discussion, the group explored ideas to improve the process that a five-member InCUS planning committee then distilled into preliminary recommendations. Throughout the in-depth process, participants recognized that much more than USMLE scoring needs to change. “Students think this is going to determine the trajectory of their careers. That’s terrifying.” Christle Nwora McGovern Medical School “The current system of applying for residencies has many parts besides the USMLE. Each was developed with good intentions, and each has evolved independently in response to environmental changes, but the combined evolution has created a system that is not serving stakeholders well. Everyone is frustrated, and no one group can fix the issues alone,” says Alison Whelan, AAMC chief medical education officer and a member of the InCUS planning committee. “We in the medical education community have a responsibility to fix these problems. We need to balance thoughtfulness and urgency both for our learners and for their future patients.” Below are insights into some of the key issues discussed, the committee’s preliminary recommendations, and a glimpse at what lies ahead. How did we get here? Today’s residency applicants often feel caught in a desperate game of musical chairs. A few statistics tell the story: Since 2010, the number of U.S. medical school matriculants has increased by 23% while residency slots haven’t kept pace. Feeling that squeeze, candidates have been submitting increasing numbers of applications. In fact, from 2014 to 2018, the average number of applications per applicant rose 15% from 79 to 91. The average number of applications from international medical school applicants rose from 119 to 136 during that period, and applications from U.S. MD-granting medical school applicants rose from 47 to 60. “We now get 1,000 applications each year and match 12 residents,” says Steven Bird, MD, program director of emergency medicine at the University of Massachusetts Medical School. “We also are faced with fewer and less objective measures of applicants’ competency. There are several reasons for that, including because many schools have gone to pass/fail, which makes it difficult to see how a candidate compares to their peers,” he says. “And schools often write such positive letters of recommendation for their students that they are borderline worthless.” That leaves programs desperate for a standardized metric to help assess applicants from a wide range of schools. USMLE scores can provide that measure. “We now get 1,000 applications each year and match 12 residents.” Steven Bird, MD University of Massachusetts Medical School But overreliance on USMLE opens a Pandora’s box of problems. For one, it places too much weight on a single measure — and one that’s not a particularly good fit for the job, many say. “This is a licensing exam that’s meant to show basic competency at the pass/fail point, but it’s being used for alternative purposes,” notes MCW’s Simons, who also chairs Wisconsin’s board of medical examiners. Excess focus on the test also can harm students’ education. “Some of my peers said, ‘I’m going to study as much as possible to prepare [for Step 1] and not go to class,’” recalls Jesse Rafel, MD, a New York University Langone Health resident who attended the conference. “That means they weren’t engaging as much in the expertise that faculty offer. And they were missing out on important areas like professionalism and communication that the test just doesn’t assess as much.” Recommendation 1: A new scoring system The first draft recommendation in the InCUS document suggests exploring changes to the USMLE scoring system. One option that could shrink student anxieties would be a switch to pass/fail grading. But without scores, programs could find their jobs even more challenging. Plus, some students, including many overseas candidates, actually prefer numerical scores. “Many students attend non-U.S. schools that may be less well-known, and they can differentiate themselves by scoring well on the test,” says Kathryn Andolsek, MD, MPH, assistant dean for premedical education at Duke University School of Medicine. “They are roughly a quarter of the U.S. resident and fellow workforce, and I’m sure that it will be much harder for them to get interviews for residencies if the test becomes pass/fail.” Pass/fail isn’t the only option, though. Another is a tiered system that could tell programs whether an applicant’s score fell into the top or bottom quartile of test-takers, for example. USMLE could also report a student’s composite grade rather than individual Step scores to decrease the focus on any one part of the test. And some conference participants suggested building something brand-new. “If we need a way to select residents, why not create a test that’s designed for that purpose, rather than using something that happens to exist?” asks Andolsek. “What we’ve been doing with the USMLE is like using a mammogram to screen for prostate cancer.” Recommendation 2: Testing the test More thorough research could help prove — or disprove — how well USMLE scores actually predict success in residency and practice, suggests the second InCUS recommendation. Certainly, researchers have already studied the link between USMLE scores and various outcomes. “We know that scores at a certain threshold level predict initial board certification,” says Michael Barone, MD, MPH, the NBME’s vice president of licensure programs. “There are also studies that link scores to outcomes in clinical medicine, but there aren’t many of these and they need more repetition with broader populations. For example, one paper showed an association between higher licensing exam scores and lower cardiovascular mortality in patients. We need more studies to understand such associations better.” Rafel says he’d like to know how useful the test is in assessing such soft skills as communication, systems thinking, and interprofessional teamwork. Still, he notes, focus on research shouldn’t supplant action. “There is concern in the community that we could get too bogged down in research,” he says. Recommendation 3: Addressing racial differences Among self-identified racial groups, research shows that white students perform higher on the USMLE than any other group. The third InCUS recommendation advised working to minimize those racial differences. “This is a complicated, nuanced problem,” says Whelan. “Individuals from lower socioeconomic backgrounds or first-generation college students, which include a larger portion of underrepresented-in-medicine students, often face structural barriers when taking high-stakes exams. Among other factors, they may not know how to access study courses or have the money for them.” USMLE already reviews test material for items that may lead to unintended group differences, Barone explains. “We don’t want the USMLE to limit diversity in medicine,” he says. “Whatever the necessary steps, the endgame is a workforce that looks much more like the U.S. population.” Recommendation 4: Healing the system “Even if we address all the issues related to the test, there are still a lot of other problems,” says Rafel. “In the end, this is a multiheaded beast.” The fourth InCUS recommendation advised creating a cross-organizational panel to tackle concerns that can hobble students’ transition from medical school to residency. “Even if we address all the issues related to the test, there are still a lot of other problems. In the end, this is a multiheaded beast.” Jesse Rafel, MD New York University Langone Health The group suggested three areas the panel could explore: reducing the number of applications candidates submit, helping programs assess applicants more holistically, and building trust in the information schools provide about their students. Reducing the number of applications If students are going to apply more sensibly and strategically, they’ll need better information about their residency options, attendees noted. “I may not know what a program truly values, such as more of a focus on research, so I might rely on word-of-mouth,” says Nwora, who chairs the AAMC’s Organization of Student Representatives. “There has to be a lot more transparency from programs if students are going to find a good fit.” Students also would benefit from more tools designed to help identify appropriate programs, participants noted. Some already exist — the AAMC’s Apply Smart, for example — but more in-depth ones are needed, such as the upcoming program comparison tool Residency Explorer. The possibilities for exploration also included a more audacious option: capping the number of applications a candidate could submit each year. “It’s hard to suggest putting a limit on anything related to someone’s career. It doesn’t seem like the American way,” says conference attendee Peter Buckley, MD, dean of the Virginia Commonwealth University Medical School. “But programs are deluged, and students face the stress, time, and expense of applying and traveling for a lot of interviews to get a residency position. No one is happy with the current system.” Supporting more holistic review Residency programs receive a lot of information about applicants — personal statements, the Medical Students Performance Evaluation, and more — but they don’t always have the tools necessary to piece together a holistic profile. One possible way forward would be to create mechanisms programs could use to sort and analyze applicants based on traits beyond scores, such as communication skills, teamwork, problem-solving, and situational judgment. Getting better information about applicants from schools The selection process — and the transition from undergraduate to graduate medical education — could be much smoother if residency programs had deeper insights into applicants, experts say. But that’s not always easy for schools to deliver. Among the challenges is ensuring that faculty can provide meaningful student assessments. “Faculty need time, support, and training so they can provide truly comprehensive assessments,” says Buckley. “That’s particularly true in the clinical setting, where doing assessments can be challenging when they are providing care, dealing with electronic health records, and more.” Simons points to another assessment concern. “Schools sometimes are conflicted. … What do you tell a program if a student has struggled a bit along the way? You care about your students, and you want them to get a job,” he says. “But of course at the end of the day, we are here to send out physicians who will serve patients well.” What happens next? During a six-week period that ended July 26, thousands of people and organizations submitted thoughts on the InCUS preliminary recommendations. Since then, committee members have been reading through comments and poring over data. Next, they’ll continue the conversation with constituents at upcoming professional events such as Learn Serve Lead 2019: The AAMC Annual Meeting. Based on all this work, they’ll determine whether the initial draft needs changes and finalize their recommendations by the end of the year. “My expectation is that the final recommendations won’t be dramatically different. If we hear or see something compelling, we will certainly take it to heart,” says Whelan. “Mostly what we’re looking for is whether we missed something really important.” “I think everyone involved cares deeply and wants to support learners. Ultimately, they want to ensure an effective and equitable process that serves the health of the nation.” Allison Whelan, MD AAMC How does the process unfold after that? Any changes related to the USMLE would need approval by the boards of the FSMB and NBME. “We’ve already started conversations with our board members,” says Barone. “They are highly engaged and value tremendously what they’re hearing from all the key stakeholders.” As for creating a cross-organizational panel to explore systemic changes, the InCUS committee is considering options. “We don’t have answers yet,” says Whelan. “We want to be very inclusive, and we need to think about what other organizations to invite.” Looking ahead, Whelan feels optimistic. “I think everyone involved cares deeply and wants to support learners. Ultimately, they want to ensure an effective and equitable process that serves the health of the nation.” Source
