It finally happened. Three months ago, the most important conference in medical education, the Invitational Conference on USMLE Scoring (InCUS), took place behind closed doors. Authors from the five stakeholders who convened the meeting—the Association of American Medical Colleges (AAMC), American Medical Association (AMA), Educational Commission for Foreign Medical Graduates (ECFMG), Federation of State Medical Boards (FSMB), and National Board of Medical Examiners (NBME)—have finally revealed what was deliberated at the meetings and made preliminary recommendations. Prior to the event, we worried that this invitational conference would not sufficiently consider the well-being and needs of medical students. We wondered whether a secretive meeting with a guest list and agenda determined by corporate entities with financial interests in the outcome could lead to meaningful change. We were afraid that any recommendations would only serve to protect the status quo and delay reform. Unfortunately, but unsurprisingly, we appear to have been right. How Proposed Changes Were Decided For those who have not closely followed the recent debate, some context may prove helpful. For years, increasing residency applications have contributed to a growing administrative burden for program directors. Unfortunately, misappropriation of USMLE scores to screen applications has turned Step 1 into a de facto "residency aptitude test." We have pointed out that the exam, intended to inform a binary determination on competency for licensure, is increasingly irrelevant to clinical practice. We have also argued that increased focus on test preparation has negative impacts on medical student well-being and institutional curricula, through the creation of an adverse "Step 1 climate." A change from Step 1's current three-digit scoring system to a pass/fail system would satisfy the test's purpose for licensure, allow students to focus on more relevant educational experiences, and facilitate holistic applicant review and reform of application inflation. To address these concerns, the USMLE's sponsor organizations (the NBME and FSMB) along with three other stakeholder organizations (AAMC, AMA, and ECFMG) organized the InCUS, which was held in March 2019. Forty-five invitees joined 20 executives from the aforementioned organizations to discuss the issues in undergraduate and graduate medical education (UME and GME) and make recommendations regarding the future of Step 1 scoring. In a podcast recorded in advance of the conference, NBME Vice President Michael Barone said, "[What] InCUS is allowing us to do is bring together all the stakeholders, a group of invitees who really are going to challenge us, who are going to ask us to look at different ways of doing things or are going to engage in thought and perhaps come to the conclusion that there are many, many changes that need to happen in that UME-to-GME transition system." This latter prediction proved eerily prescient when the preliminary recommendations from the conference were released. Disheartening Recommendations Given the complexity of the issues involved, achieving a consensus was always likely to be difficult. However, the recommendations are nonetheless surprisingly sparse and nonspecific. Rather than making a specific recommendation on USMLE score policy, the report recommends "consideration of changes such as pass/fail scoring." The document mentions other possibilities, such as switching to a tiered scoring system. Examples cited included "quartiles, quintiles, or some other division." Another suggested possibility is creating a composite score that consists of "aggregate performance information from Step 1, Step 2 CK, and Step 2 CS." Such solutions seem unlikely to diminish the outsized roles that these examinations play in residency selection or students overemphasizing examination preparation. The second InCUS recommendation seems laudable at first glance: to "minimize racial demographic differences in USMLE performance." Step 1 has been shown to play a role in perpetuating disparities as well as racial bias in medical education for underrepresented minorities in medicine. Few would argue with the intent of this recommendation. However, the steps that might actually be taken to achieve this goal are left largely unexplored. The USMLE simply states that it will "prioritize this line of work." The final two recommendations laid out by the conference participants are to "accelerate research on the correlation of USMLE performance to measures of residency performance and clinical practice" and to "convene a cross-organizational panel to create solutions for challenges in the UME-GME transition." The first suggestion seems unnecessary in light of the existing literature. Ample evidence demonstrates that although Step 1 scores have some use in predicting performance on other standardized exams, their correlation with skills valued by patients is poor or nonexistent. Although a cross-organization panel to discuss problems in the residency application process might prove beneficial, it could also be viewed as a way to delay a decision and maintain the status quo. This is especially true when considered in the context of the stakeholders' prior reluctance to consider changes to the Step 1 scoring system. Looking at the four recommendations as a whole, we are disheartened. Rather than offering clear solutions and taking the views of medical students and their advocates into account, the InCUS recommendations seem to echo a desire to avoid disruption. However, by avoiding this disruption, InCUS has also avoided leadership. By pointing out that the problems with Step 1 are just one problem in the UME-GME transition, they encourage a kind of circular logic in which maintaining the status quo is inevitable. Making Step 1 pass/fail is not a cure-all, but it is a necessary first step in bringing reform to the many problems the conference attendees highlighted. Rather than kicking the can down the road with more meetings, we had hoped for concrete solutions to the current crisis in medical education brought on by the overuse of the three-digit Step 1 score in the residency selection process. Ultimately, InCUS failed to deliver an actionable plan to achieve any goal. In so doing, it again calls into question the ability and desire of the convening organizations to enact meaningful change. An opportunity for medical students and their advocates to force change on this issue may still exist. The organizations that convened InCUS are seeking public comments until July 26, 2019. The medical testing establishment has shown that it is not exactly excited to embrace changes to a status quo that significantly financially benefits them. However, if we work together, we may be able to generate enough pressure to force a change that will ultimately benefit our future doctors and their patients. The views expressed are the authors' own and are not necessarily representative of the policies or positions of their respective institutions. Source
