Two days after they announced the second cancellation of scheduled exams this year, the ABR felt compelled to rapidly address the massive discontent in the radiology committee. Perhaps hearing that a joint letter from every trainee and many rad organizations was in the pipeline gave them the extra incentive to try and preempt (unsuccessfully, see below) formal censure from its stakeholders.

On the whole, ABR President Vincent P. Mathews’ letter is…defensive. And it addresses the core concerns by calling them out but largely without meaningful explanation that would make the ABR’s repeated unforced errors in the handling of the pandemic look like anything other than inept and out of touch. It sounds like he’s inherited this mess, but the ABR needs more than a bigger listening ear: They need a paradigm shift.

I’ve been asked if I could translate from ABR-speak into English, so without further ado.

Colleagues,

Based on feedback received after the ABR’s announcement that we are canceling most of our exams for 2020 due to the COVID-19 pandemic, I want to provide more information regarding the considerations that led to this decision.

Restrictions on large gatherings in many locations, including Chicago, where we have our largest testing center, are very unlikely to be lifted before the fall. In addition, our staff had thoroughly investigated the logistics required to deliver an in-person exam, and our conclusion was that we could not guarantee the safety of our candidates, diplomates, and staff to the level required to proceed.

We know everyone told us that planning for an in-person exam in November was nuts. We know there was never any chance the virus would magically disappear or that chilly Chicago in November would ever be a better place for humans than June let alone a safer place for virus-free travel. I know people told us it never had a chance of happening and that if somehow we did have the ability to force people to come that it would be a disaster. Instead of listening, we wasted valuable time that we could have spent preparing a remote solution trying to figure out a way to use the testing centers we should have never built.

Many continue to ask about giving our exams at local testing centers. We do offer some of our Radiation Oncology and Medical Physics Exams at Pearson VUE centers, and we still have some scheduled in December. Neither Pearson VUE nor Prometric has the technical capability to deliver our Diagnostic Radiology Exams and have not been interested in developing those capabilities when we have inquired repeatedly over several years. Currently, these centers are closed, and it is uncertain when and where they will open.

A long time ago when we developed these exams we didn’t take into account the available capabilities of commercial centers (though we haven’t and have no intention of ever disclosing what those missing capabilities are or why we didn’t just go back and pay some nerds to re-make the software to make this feasible). After all, a bunch of our own exams as well as those offered by the American Board of Nuclear Medicine and the ACR In-service are offered at places like this. In the end, building our own centers so that we could reduce the ambient light by a few lux seemed like an easier and more profitable solution.

Others have asked why we can’t deliver a virtual exam. We are actively exploring alternatives and are collaborating with other ABMS boards to evaluate options. Currently, we have no reliable, secure option for a high-stakes exam such as the initial certifying exams that we deliver. We are aware that the American Board of Surgery and the American Board of Ophthalmology have announced they will give virtual oral exams in 2020. Their exams are different than ours, and they have not validated the ability to use radiologic images in those exams. No board has developed a secure initial certification written exam virtual platform that would be needed to deliver our Diagnostic Radiology and Interventional Radiology Core and Certifying Exams. When the breast content on the core exam was not successfully delivered to all candidates in 2017, we devised a solution for individuals to take that content remotely rather than come back to the testing centers. During this administration, some candidates had technical problems that we were able to fix “on the fly,” which was possible because of the small number of candidates taking the short exam at any given time. That exam contained very limited content and, in fairness to the candidates, we removed some items that were potentially too subtle without the monitor display control we have in our testing centers. Fortunately, it worked for a session of less than an hour for a minority of our candidates; however, we are not confident that it would work for a 10-hour exam for more than 1,000 candidates. Because of the magnitude of the significance of our initial certification exams, we must ensure near complete reliability. As we have experienced on two occasions in the past few years, even a minor failure in the delivery of these exams is extremely disruptive to everyone involved.

I know that no one actually expects us to have a ready-made, rapidly-deployable remote solution, but I feel compelled to conflate the fact that that we currently lack this capacity with our general unwillingness to pursue this goal with full vigor and the financial might of our near limitless cash reserves.

Our exams aren’t like other exams. Each exam is truly unique, like a snowflake.

We know that our continued harping on “image quality” is tone-deaf and completely meaningless given the functionality of literally every computing device used over the past twenty years, the existence of image-rich exams in multiple settings including our very own OLA, and the fact that cross-sectional modalities use images that could have been displayed by a Super Nintendo.

We also know that trying to crap on the “reliability” of our one-time remote experiment may also fall flat given that said effort came because our home center was even less reliable in the first place.

These mental gymnastics may be nonsensical, but they’re our party line, and they’re the best we got.

To reiterate, we just don’t have a reliable platform to give our computerized or oral exams right now. We are exploring our options for each of our specific types of exams in our various specialties and, hopefully, we can find solutions in the coming months. Programs have been asking for a decision soon because they are creating their resident schedules for the next academic year; therefore, we decided to move ahead with our announcement. We know that it will be highly disruptive to have two years of residents taking the Diagnostic Radiology and Interventional Radiology/Diagnostic Radiology (IR/DR) Core Exam in June 2021. Therefore, we hope to offer an administration in Chicago and Tucson in February 2021 in the event we do not have a virtual option. We are finalizing the dates and will make an announcement as soon as possible. We will also engage program directors and others as we progress with any virtual solutions.

We are looking, but we don’t want to promise to do the right thing. We’d rather arbitrarily postpone the date to another fictional timepoint instead of promising to carry out a remote solution on the dates we already provided and for which all programs and individuals have since prepared.

That said, we know—deep down—that we must provide a remote testing solution. And we know, since commercial centers won’t be able to take on this business, that it must either involve programs/institutions or people at home. But we are simply not ready to admit this to ourselves or others except in the most oblique of ways.

This is another unfortunate situation created by an unprecedented pandemic, and I hope you understand that this decision was not made lightly. We are fully aware of the disruption this causes for our candidates and their programs and desperately would have liked to have an acceptable solution sooner. We take responsibility for not engaging more stakeholders in the conversation leading to this decision, and we commit to doing better in this regard. Over the next two years of my service as ABR President, with the full support of the entire organization and Board, I will make this a priority. However, I do not think we would have arrived at a different outcome at this point in time.

We know that announcing a plan isn’t the same thing as having a solution in place, but we hope you understand that we believe all the current drama is entirely secondary to an unavoidable pandemic and not at all a reflection of our systemic deficiencies. We are sorry that we didn’t listen to your concerns formally before ignoring them. We promise to have an increased number of superfluous token conversations in the future.

And so

This may have delayed the formal response from a ton of radiologists by a day or two, but it changed nothing:

Together, 17 national radiology groups urge the @ABR_Radiology to offer virtual remote or in-person local/regional administrations of the Core, Certifying, and Subspecialty Exams in Fall 2020. #HearUsABR. pic.twitter.com/3hPrvtNEyo

— MERCCoalition (@MERCCoalition) June 8, 2020

It’s a good letter, and you should read it. The joint effort covers the salient points in measured tones and doesn’t once devolve into the ruthless mockery that less restained commentators might have been unable to resist.

Yesterday, the American Board of Radiology announced in an email blast that it was canceling the planned in-person administrations of the Core and Certifying Exams and postponing until an unspecified time in 2021.

I wasn’t the only person to tell the ABR that its November in-person exam plan was–from its inception–magical thinking. So on one hand, the announcement was long overdue. On the other, they’re still missing the point: the only solutions to the problem of high-stakes medical exams right now are disseminated ones, and the number of other boards moving to virtual exams continues to grow steadily. Centralized nationwide travel is simply a nonstarter, and the delays here only serve to lengthen the duration of the ABR’s missed opportunity to do the right thing.

Unlike the June cancellation, this time the ABR gave no date. That’s because any date would be a meaningless guess. There is no end in sight for this pandemic. We have no idea when it will be appropriate for nationwide travel for something as stupid as a multiple-choice test. There is a highly nonzero probability that anything of the sort in 2021 will remain unconscionable. So while they caved to the reality that the November date was untenable, they won’t acknowledge the deeper reality that their testing centers became obsolete forever earlier this year.

The vague notion of a “2021” administration is a plan without a plan. Programs can’t meaningfully change rotation and call schedules for a second time to a random new date. Residents trying to account for these high-stakes exams in their family planning are once again left helpless. Instead, everyone will need to contend with a combination of two classes of residents taking the exam in rapid succession or even at the same time. This is a completely avoidable logistical disaster for training programs.

Remote Testing: The Obvious Solution

Here is what the ABR said about remote testing in their email:

In response to numerous queries, delivery of the [Core and Certifying Exams] using available remote platforms is not practicable. We have investigated many options, but the inability to adequately control image quality, the testing environment, and security would significantly threaten the fairness, reproducibility, validity, and reliability of the testing instrument across all candidates.  We want to give an exam that will accurately reflect the hard work of our candidates on their path to certification.

Candidates and trainee advocates deserve to know why. They deserve to know what those investigations were and why the ABR feels no solutions can be used. They deserve to know why if “available” options won’t work why the ABR will not create their own. After all, they created their own test and built their own testing centers! And then, when one of those centers failed in 2017, they even managed to administer part of that exam remotely to candidates at home! And finally, we all deserve to know why the ABR changed from using practical to practicable in its correspondence.

It’s completely insufficient to just say it’s not practicalpracticable to do a remote solution.

Take ownership of this communication.

Literally–and I mean literally–no one believes you. A simple “it’s not going to happen” is not a suitable justification for the continued disruption, inconvenience, cost, and health concerns of an in-person exam.

The current exam set up was suboptimal for years, but now it’s inappropriate.

To be fair, I believe the ABR when they say that commercial testing centers are not a viable option right now. Even the NBME has been forced to enroll medical schools to help administer the Step exams given issues with Prometric during the pandemic. These places are canceling exams left and right in order to maintain a semblance of social distancing, but even on a good day, these companies provide a poor examinee experience. It’s 2020 and we can do a good job with exam security without subjecting candidates to the degrading experience of the Prometric pat-down. (On a related note, this would be a great time for the ACR to roll out a program-administered in-service exam to demonstrate how to do this to the ABR).

Regardless, it’s possible to do live-proctoring via webcam. It’s possible to do live-proctoring in-person at residency programs. There are multiple options to administer an exam that may be harder than “flipping a switch” (their words) but are still solutions worth actively pursuing. I wrote about some of these common-sense approaches in April.

A Pathologically Absent Sense of Irony

The ABR says that remote testing cannot happen because it would prevent them from ensuring the “reproducibility, validity, and reliability of the testing instrument across all candidates.”

But these delays have already ruined this exact thing, and further delay will only do so more. While these exams are theoretically criterion-referenced by the Angoff committees, at the end of the day, the Core exam is an exam taken by rising senior residents in June after a set three-year period and an established curriculum. Every class is measured at the same time in the same place.

The ABR is focused on the physical setting, but they are ignoring the timing.

For the residency class of 2021, it will be an exam taken by senior residents deep into their final spring, potentially at the very end of their training (or yes, if we’re being honest, during fellowship). There will be residents who may have begun subspecializing via mini fellowships. For ESIR residents, they may be months from their last diagnostic rotation.

If the Core Exam really tested things that every radiologist should know, then perhaps these seasoned seniors would pass at rates higher than ever before. But as those who have taken the Core Exam already know, that isn’t the case. These seniors will need to study hard (again) to maintain the superfluous working-knowledge to succeed over the additional months of waiting time. When they sit down for the exam in Chicago or Tuscon, they will be exhausted in addition to bitter, angry, and maybe even scared. They will be wearing masks and trying to stifle errant coughs. It will be a miserable and unnecessary experience.

Between the delay and the atmosphere, this will never be the same instrument. But the beauty of the ABR’s inscrutable system is that their answer to any grading questions will always be trust us.

Endlessly repeating a presumably earnest desire for reproducibility, validity, and reliability is the best the ABR can muster, but it reminds me of recent Trump comments about mail-in ballots: tangentially related to a real concern but ultimately hysterical, impotent, and rooted in fear.

Follow the Money

The gut reaction of everyone is that the ABR won’t do remote testing because they want money. And I want to push back on that a little because it’s an oversimplification. It’s not just about money.

The ABR will take your fees regardless of how they run the exam. Yes, they’ve invested in developing and running their own testing centers, but those empty rooms gobble up money whether they’re being used or not. Some of their reluctance may have something to do with the duration and terms of their leases, but they aren’t sharing. Ultimately, that’s a sunk cost no matter what the future holds. While it would certainly cost some money to develop a way to share the exam widely (which they may have already done the heavy lifting for), they wouldn’t lose much money upfront if exams were taken at residency programs or at home this year.

The big problem with a remote test is that it’s a long term one-way street. The ABR has maintained for years that it alone could administer these exams. But if we can have a remote exam now, then that position is simply not true, which means they’ve either been wrong for all those years, or worse, lying. I don’t think we should discount their reticence to own up to past mistakes. You don’t have to watch the news often to know that doubling down against an inconvenient truth is common practice. This was a strategic error from the day it was announced.

If the ABR does eventually use a commercial center, for example, then yes, some of those fees will go to the center, and the ABR will lose some profits. For example, this random Prometric document says they charge $105 for 8-hours of seat time (to be clear I have no idea if this is accurate). A trainee will pay 5 years of $640 fees to take the Core and Certifying Exams, so currently $3200. If we estimate that the three total days of exam time (2 Core, 1 Certifying) would cost in the neighborhood of $315, then just under 10% of your total fees would be lost. This is what sacrificing your health and convenience is worth to the ABR.

Can the ABR deal with that financial hit? Well, yes, of course. They are a multiple-choice question development company primarily staffed by volunteers with $50 million in the bank.

But if the ABR were to be true pioneers here and permanently move to program- or home-based testing, then there would be little financial hit at all.

A Sad Reality

Ultimately this is about optics and control, not about practicalities practicabilities. The ABR–entrenched in a decade-old decision to manage their own testing centers and now doubling down on their no-remote party line during the pandemic–is just doing the unequivocally wrong thing.

It seems that $50 million in cash reserves is insufficient financial lubrication to help them move past their prior strategic missteps because make no mistake: this is truly a psychological–not practicable–barrier. Shame is a powerful motivator for cognitive dissonance.

Every resident, fellow, and advocate should engage with every single organized radiology society to sign a unified letter of concern. It may not matter, but the ABR should know they stand alone.

From ABR President Brent Wagner’s article, “ABR, Stakeholders Remaining Flexible During Uncertain Times,” in the May issue of the BEAM:

For the ABR, as it became clear that standard exam development activities would not be effective in the short term, staff and volunteer content experts quickly made adjustments: test assembly activities for upcoming exams were revised to use remote conferencing software. While this is often not optimal for the volunteers, the modification to the process promised to support continued success in exam creation that would be relevant and cost effective.

The real stakeholders here are the residents who should be taking a live-proctored test from the comfort of their home next month.

The volunteers who do the heavy lifting of test creation may have been given the flexibility to volunteer from home, but so far the ABR has been unwilling to “remain flexible” enough to do the right thing and make remote testing a reality like the American Board of Surgery recently successfully demonstrated. Even the NBME has temporarily canceled USMLE Step 2 CS with the understanding that nationwide travel is simply inappropriate for a healthcare-related examination involving scores of doctors for the foreseeable future.

Some financial considerations are almost certainly tied to this desperate need to have a live in-person exam that no one has ever wanted, but departing highly-compensated Executive Director Valerie Jackson said this in her departing missive:

Another big part of my life has been volunteering. Over the years, some of my friends have wondered why I would give so much of my time for no pay. The reward isn’t monetary; better than money for me was the sense of giving back, the ability to work with wonderful people who are now long-term friends, and having an impact on many facets of our profession.

I have no words. Total compensation in 2018? $834,567.

I did, however, like the final paragraph of incoming ABR President Robert Barr’s entry:

We still have work to do. We need to keep costs down, we need to ensure that testing is meaningful and accurate, and we need to improve the overall test experience for our candidates, among other challenges. We are working on all of these today. I hope you’ll give us a chance, and I welcome your suggestions and constructive feedback.

I would love nothing more than for the ABR to pivot and start making the correct operational and strategic decisions to succeed in those areas. The ACR recently made some suggestions.