The Coming Changes to USMLE Scoring

In March of this year, there was the InCUS: Invitational Conference on USMLE Scoring. The results page is here, and the summary report is here.

Invitational? That means that the only people invited were stakeholders who are deeply entrenched in the status quo and/or directly profit from the USMLE system. Namely, the Association of American Medical Colleges (AAMC), American Medical Association (AMA), the Educational Commission for Foreign Medical Graduates (ECFMG), the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME).

Absent? Regular humans like students, residents, or even much in the way of program directors, educators, etc. No big surprise. When a growing body of students and educators advocated for removing Step 2 CS because it was an easy superfluous reduplicative and expensive waste of time, the NBME just made that harder to pass. They’d much rather just change the paper you get at the end of the other tests than introspect or make a structural change.

So, the NBME has always said they didn’t like people using the score to evaluate medical students (but have spent an awfully long time letting people do just that):

Said another way, the exams were developed as medical licensure examinations and not as academic achievement exams.

The outcome of this big convening of masterminds? Well, the recommendations are extremely vague but give the impression that eventually removing the three-digit USMLE score is a likely component.

Recommendations specific to USMLE:
1) Reduce the adverse impact of the current overemphasis on USMLE performance in residency screening and selection through consideration of changes such as pass/fail scoring.
2) Accelerate research on the correlation of USMLE performance to measures of residency performance and clinical practice.
3) Minimize racial demographic differences that exist in USMLE performance.

Recommendations to the UME-GME transition system:
1) Convene a cross-organizational panel to create solutions for the assessment and transition challenges from UME to GME, targeting an approved proposal, including scope/timelines by end of calendar year 2019.

Indeed.

One of the unintended consequences of more medical schools moving to pass/fail amidst increasing medical school enrollment and flat residency spot numbers has been the increasing importance of the USMLE and the shadow curriculum it has created.

If Step 1 matters but your coursework does not, then you’d be better off in a correspondence course that let you spend all your time preparing for Step 1 and ignoring anything your school actually wants you to do. On the flip side, if the USMLE were to suddenly be pass/fail, then residency programs may be evaluating applicants with literally no comparative data from which to judge candidates.

Point #2 from the blockquote is fascinating in its awkward tardiness because everyone knows the correlation with most clinical performance is negligible, and no meaningful research would likely ever state otherwise. USMLE scores correlate with written boards’ pass rates, which themselves also do not correlate with clinical performance. It’s turtles all the way down. None of these tests actually test what they purport to. The whole system is in shambles from the SAT on up. They all measure a degree of general intelligence and preparation, but…who cares.

Despite the mea culpas about mental health, failing students, blah blah blah, not discussed at all–of course–is whether or not the USMLE sequence should even be maintained as is. There’s a painful failure of vision in a conference solely focused on…scoring.

For example, is CS something the NBME should be doing in the first place or isn’t that what an accredited medical school is for? Or, are Step 1, 2, and 3 testing sufficiently different things to do justify three different exams, and if they are, do all three really play a distinct role in the licensing process? One could easily argue that Step 2 CK is much more meaningful to clinical practice and residency performance than Step 1, which mainly has the benefit of a) being hard and b) having scores universally available during the residency application process because it’s taken earlier.

Feel free to submit your comments on these meaninglessly vague preliminary recommendations here.