Lessons from the American Board of Surgery’s Virtual Board Exam Debacle

Lesson 1: Beware Third Party Services

During the slow drama that has been the delivery of high-stakes during the pandemic, some organizations like the American Board of Radiology resisted agreeing to a virtual exam for so long precisely because they couldn’t control the environment, delivery, and security. Unfortunately, we have some great evidence that offloading exam security to a third-party company can easily fail on those metrics as well.

Just ask the American Board of Surgery, which hired Verificient’s Proctortrack, an “industry leading Online Proctoring / Remote Invigilation solution.” Leaving aside how creepy “invigilation solution” sounds, Proctortrack has a 1-star rating on the app store for being generally awful. And despite the platform’s buzzwords of AI, computer vision, and machine learning, there were both serious mechanical and human failings.

The ABS went from a single day 8-hour exam to a two-day 4-hour exam, just to make sure the platform could handle it. It couldn’t.

ProtorTrack is extremely invasive. It requires download and installation of monitoring software that gives serious access over both your computer and your smartphone. The service also requires a room scan of 360 degrees to clear your workspace of nefarious intent (that last part actually makes sense to me).

So what happened?

Near instantaneous failure. The Room Scan feature failed early, and then examinees were basically left out to dry with poor phone and online customer service. People kept trying to log in and ended up banging their heads against the wall. I’ve heard an estimate that around half of the residents were able to take at least a portion of the test. But the circumstances were less than ideal, with many residents apparently reporting being harassed to complete distraction by the “live” overseas proctors.

Shortly after, the reports of identity theft, unauthorized credit card charges, and inappropriate social media contact began.

Because the test questions were exposed to a fraction of the examinees, those questions are no longer usable. The ABMS boards talk a lot about “their” intellectual property, and while these questions are written by volunteers, we shouldn’t demean the serious effort that goes into crafting and then vetting them. MCQ tests may not be a great measure of medical competence, but that doesn’t mean a lot doesn’t go into making one. Having to throw out a test is a serious setback.

I don’t know what goes into how these organizations make these sorts of operational decisions, but I don’t think it takes a clairvoyant to predict that ProctorTrack might have been a suboptimal choice.

While it might look good to an uninvolved observer, I don’t think a third-party off-site unaccountable organization is the right move for a high-stakes virtual exam. While at home exams require more effort, a residency-based in-person exam with live or focused virtual proctoring would be relatively straightforward. Regardless, video proctoring and screen recording don’t even need to be live, so long as all information is saved and can be reviewed.

It’s also hard to imagine a doctor throwing away their career just to copy some exam questions, which is really what we’re worried about here more than someone cheating to pass.

Lesson 2: Be Flexible

The ABS essentially forced residents to take the test (before they canceled it and told them they couldn’t). Taking a pass this year would use up an attempt and looks bad for programs.

One one hand, that doesn’t sound so bad if it all worked out. But it didn’t. And when things don’t work out, people’s lives are way harder. Dedicated study time, job starts, family planning–everything. Doctors build their lives and programs plan their schedules around these exams. When we live in tough times, it’s best to err on the side of accommodation.

Any board needs to have contingency plans in place that will accommodate a rapid administration to waiting candidates. Write more questions now and have them ready to deploy. Running out of questions after a failed attempt and needing to push things back indefinitely in order to write more is a terrible plan.

Lesson 3: Prioritize Communication

Announcements about the exam were done primarily through Twitter. Emails and website updates came universally after a multi-hour delay. Twitter is fine, but Twitter shouldn’t be used as a replacement for individualized communication. When it comes to a career-defining exam, people should be getting so many emails (or potentially opt-in text messages) that they almost find it annoying.

By the time the ABS finally published a FAQ on its website, days had passed.

When the Resident Association of the American College of Surgeons (RASACS) and ABS hosted a virtual town hall, it was announced within five hours of the event and then limited to in-advance sign-ups of only 400 people, a small fraction of the affected residents.

Take-Home

I don’t want to suggest that doing a virtual exam for the first time in history is a straightforward endeavor.

What I do think is clear is that an organization whose sole purpose is to create and administer tests needs to be nearly flawless on that execution. Offloading tasks to third parties, even the commercial testing centers so commonly used, has long resulted in a suboptimal and often degrading experience. Third-party proctoring is apparently no different. I appreciate that ABMS member boards are not tech companies and will always use contractors to accomplish a variety of tasks, but that does not absolve them of responsibility for the final product.

Lastly, communication and accountability shouldn’t be an afterthought. Just because residents have no power over their respective boards doesn’t mean they aren’t worthy of consideration and respect.