The ABR and the Practicability of Doing Its Job

Yesterday, the American Board of Radiology announced in an email blast that it was canceling the planned in-person administrations of the Core and Certifying Exams and postponing until an unspecified time in 2021.

I wasn’t the only person to tell the ABR that its November in-person exam plan was–from its inception–magical thinking. So on one hand, the announcement was long overdue. On the other, they’re still missing the point: the only solutions to the problem of high-stakes medical exams right now are disseminated ones, and the number of other boards moving to virtual exams continues to grow steadily. Centralized nationwide travel is simply a nonstarter, and the delays here only serve to lengthen the duration of the ABR’s missed opportunity to do the right thing.

Unlike the June cancellation, this time the ABR gave no date. That’s because any date would be a meaningless guess. There is no end in sight for this pandemic. We have no idea when it will be appropriate for nationwide travel for something as stupid as a multiple-choice test. There is a highly nonzero probability that anything of the sort in 2021 will remain unconscionable. So while they caved to the reality that the November date was untenable, they won’t acknowledge the deeper reality that their testing centers became obsolete forever earlier this year.

The vague notion of a “2021” administration is a plan without a plan. Programs can’t meaningfully change rotation and call schedules for a second time to a random new date. Residents trying to account for these high-stakes exams in their family planning are once again left helpless. Instead, everyone will need to contend with a combination of two classes of residents taking the exam in rapid succession or even at the same time. This is a completely avoidable logistical disaster for training programs.

Remote Testing: The Obvious Solution

Here is what the ABR said about remote testing in their email:

In response to numerous queries, delivery of the [Core and Certifying Exams] using available remote platforms is not practicable. We have investigated many options, but the inability to adequately control image quality, the testing environment, and security would significantly threaten the fairness, reproducibility, validity, and reliability of the testing instrument across all candidates.  We want to give an exam that will accurately reflect the hard work of our candidates on their path to certification.

Candidates and trainee advocates deserve to know why. They deserve to know what those investigations were and why the ABR feels no solutions can be used. They deserve to know why if “available” options won’t work why the ABR will not create their own. After all, they created their own test and built their own testing centers! And then, when one of those centers failed in 2017, they even managed to administer part of that exam remotely to candidates at home! And finally, we all deserve to know why the ABR changed from using practical to practicable in its correspondence.

It’s completely insufficient to just say it’s not practicalpracticable to do a remote solution.

Take ownership of this communication.

Literally–and I mean literally–no one believes you. A simple “it’s not going to happen” is not a suitable justification for the continued disruption, inconvenience, cost, and health concerns of an in-person exam.

The current exam set up was suboptimal for years, but now it’s inappropriate.

To be fair, I believe the ABR when they say that commercial testing centers are not a viable option right now. Even the NBME has been forced to enroll medical schools to help administer the Step exams given issues with Prometric during the pandemic. These places are canceling exams left and right in order to maintain a semblance of social distancing, but even on a good day, these companies provide a poor examinee experience. It’s 2020 and we can do a good job with exam security without subjecting candidates to the degrading experience of the Prometric pat-down. (On a related note, this would be a great time for the ACR to roll out a program-administered in-service exam to demonstrate how to do this to the ABR).

Regardless, it’s possible to do live-proctoring via webcam. It’s possible to do live-proctoring in-person at residency programs. There are multiple options to administer an exam that may be harder than “flipping a switch” (their words) but are still solutions worth actively pursuing. I wrote about some of these common-sense approaches in April.

A Pathologically Absent Sense of Irony

The ABR says that remote testing cannot happen because it would prevent them from ensuring the “reproducibility, validity, and reliability of the testing instrument across all candidates.”

But these delays have already ruined this exact thing, and further delay will only do so more. While these exams are theoretically criterion-referenced by the Angoff committees, at the end of the day, the Core exam is an exam taken by rising senior residents in June after a set three-year period and an established curriculum. Every class is measured at the same time in the same place.

The ABR is focused on the physical setting, but they are ignoring the timing.

For the residency class of 2021, it will be an exam taken by senior residents deep into their final spring, potentially at the very end of their training (or yes, if we’re being honest, during fellowship). There will be residents who may have begun subspecializing via mini fellowships. For ESIR residents, they may be months from their last diagnostic rotation.

If the Core Exam really tested things that every radiologist should know, then perhaps these seasoned seniors would pass at rates higher than ever before. But as those who have taken the Core Exam already know, that isn’t the case. These seniors will need to study hard (again) to maintain the superfluous working-knowledge to succeed over the additional months of waiting time. When they sit down for the exam in Chicago or Tuscon, they will be exhausted in addition to bitter, angry, and maybe even scared. They will be wearing masks and trying to stifle errant coughs. It will be a miserable and unnecessary experience.

Between the delay and the atmosphere, this will never be the same instrument. But the beauty of the ABR’s inscrutable system is that their answer to any grading questions will always be trust us.

Endlessly repeating a presumably earnest desire for reproducibility, validity, and reliability is the best the ABR can muster, but it reminds me of recent Trump comments about mail-in ballots: tangentially related to a real concern but ultimately hysterical, impotent, and rooted in fear.

Follow the Money

The gut reaction of everyone is that the ABR won’t do remote testing because they want money. And I want to push back on that a little because it’s an oversimplification. It’s not just about money.

The ABR will take your fees regardless of how they run the exam. Yes, they’ve invested in developing and running their own testing centers, but those empty rooms gobble up money whether they’re being used or not. Some of their reluctance may have something to do with the duration and terms of their leases, but they aren’t sharing. Ultimately, that’s a sunk cost no matter what the future holds. While it would certainly cost some money to develop a way to share the exam widely (which they may have already done the heavy lifting for), they wouldn’t lose much money upfront if exams were taken at residency programs or at home this year.

The big problem with a remote test is that it’s a long term one-way street. The ABR has maintained for years that it alone could administer these exams. But if we can have a remote exam now, then that position is simply not true, which means they’ve either been wrong for all those years, or worse, lying. I don’t think we should discount their reticence to own up to past mistakes. You don’t have to watch the news often to know that doubling down against an inconvenient truth is common practice. This was a strategic error from the day it was announced.

If the ABR does eventually use a commercial center, for example, then yes, some of those fees will go to the center, and the ABR will lose some profits. For example, this random Prometric document says they charge $105 for 8-hours of seat time (to be clear I have no idea if this is accurate). A trainee will pay 5 years of $640 fees to take the Core and Certifying Exams, so currently $3200. If we estimate that the three total days of exam time (2 Core, 1 Certifying) would cost in the neighborhood of $315, then just under 10% of your total fees would be lost. This is what sacrificing your health and convenience is worth to the ABR.

Can the ABR deal with that financial hit? Well, yes, of course. They are a multiple-choice question development company primarily staffed by volunteers with $50 million in the bank.

But if the ABR were to be true pioneers here and permanently move to program- or home-based testing, then there would be little financial hit at all.

A Sad Reality

Ultimately this is about optics and control, not about practicalities practicabilities. The ABR–entrenched in a decade-old decision to manage their own testing centers and now doubling down on their no-remote party line during the pandemic–is just doing the unequivocally wrong thing.

It seems that $50 million in cash reserves is insufficient financial lubrication to help them move past their prior strategic missteps because make no mistake: this is truly a psychological–not practicable–barrier. Shame is a powerful motivator for cognitive dissonance.

Every resident, fellow, and advocate should engage with every single organized radiology society to sign a unified letter of concern. It may not matter, but the ABR should know they stand alone.