Many months into the pandemic, and we’re all acquainted with the difference between a true public health measure and security theatre. Being outdoors instead of crowding inside? Meaningful intervention. Daily temperature screens? Theatre. We know that most people with the virus who are putting themselves around other humans will not be actively febrile but are still capable of spreading it. These measures are designed to make you feel better about engaging in an activity or to preserve the pretense of control in an ultimately uncontrollable scenario.
And so it is with remote testing.
For as long as there have been high-stakes exams, there has been high-stakes exam security. No student is a stranger to a live proctored exam, and we are all familiar with the commercial testing centers and their uncomfortable low-budget airport security facsimile. You would be forgiven for assuming that these measures were all to prevent cheating, and that is certainly part of the purpose, but individual dishonesty on a big exam ultimately isn’t the most pressing concern: it is the control of intellectual property. These exams cost money and time to create, and having the questions widely shared by some intrepid thief invalidates them and makes development more difficult and expensive.
Some organizations, like the National Board of Medical Examiners, increased testing capacity during the pandemic by expanding live proctoring to include selected medical schools. This made a lot of sense because medical schools give tests all the time and have the resources and space that can be easily utilized for exam administration.
Other organizations have looked to employ third-party online virtual proctoring solutions for exam security. An example of these services would be ProctorTrack, the company at the heart of the massive failure of the American Board of Surgery’s attempted virtual board examination this summer.
My board, the American Board of Radiology, has announced its intention to use a similar service, though they haven’t specified the details.
Third-party proctoring
I’m going to argue that expensive and invasive monitoring solutions like ProctorTrack sacrifice a lot in personal security and inconvenience for a modest benefit.
To proctor a high stakes exam, what you really need is a webcam turned on with both real-time and recorded video and audio of the examinee. You need to be able to watch their behavior as they take your exam, and you should be able to interact with them by audio if needed. This is enough to discourage and catch a casual cheater.
But what about the industrious premeditated antiestablishment cheater hell-bent on copying the test and then releasing it like Wikileaks? Well, the solution these platforms have for that is a combination of electronic control and visual monitoring. That starts with control of your phone and control of your computer so that you can’t run non-sanctioned software and all your actions are recorded. They usually employ some sort of “Roomscan,” which is what ProctorTrack called their “AI-powered” environment screening feature to supposedly able to capture security contraband.
I don’t know what nonsense data these companies use to train their algorithms, but let’s just be reasonable and agree that no 360-degree video sweep is going to see through semi-opaque objects, under all surfaces, or check for hidden pocket sewn into the crotch of your pants.
So yes, if IP theft is a low-hanging fruit type of crime, then these measures raise the bar. But most people aren’t going to cheat, and anyone truly determined to likely still can. The downsides of security theatre are very real: personal insecurity and platform instability. The American Board of Surgery got to experience both as candidates began receiving Facebook friend requests from proctors and seeing unauthorized credit card charges almost as fast as exam administration was canceled.
Alternative solutions
I don’t mind if people agree to a third-party proctoring platform if it at least works, but I would argue this sort of invasiveness should be an option and not a requirement. A live and record video-proctored exam with software that limits the most egregious forms of screen recording etc (similar to that used by UWorld, for example) would be reasonable.
The ABR should also offer disseminated in-person testing in a relatively safe controlled environment like at a residency program, which would allow the ABR to rely on a combination of live proctoring by residency programs and/or ABR volunteers as well as remote first-party proctoring themselves.
The Best Solution
The real solution, ultimately, is to have an exam that does not rely on gotcha questions to test raw medical knowledge. As long as the ABR focuses on multiple-choice questions coupled with an image or two, the exam will remain vulnerable to question theft and recalls.
Virtual proctoring or not, every medical MCQ exam has already created a robust industry of question bank products peddling glorified recalls. The solution won’t be found in the monitoring but in the test development itself.
The lion’s share of a radiology certification exam should be the internet-enabled practice of radiology. Being able to accurately interpret real exams, as I’ve argued before, is by far the best testing format: high fidelity simulation has not just the best face and construct validity but almost certainly the best content and predictive validity as well.
If it’s not a PACS, then it’s probably not a good test.
One Comment