Ben White

ABR totally botches 2017 Core Exam

06.27.17 // Radiology

This email belies how royally the ABR botched the 2017 Core Exam.

What the ABR should have done is what any accountable organization should do when they mess up.

  • Express regret and acknowledge responsibility
  • Be transparent and describe the mistake
  • Give an action plan and steps to correct the problem
  • Ask for forgiveness

Instead, examinees received the lip service version.

“Technical issue” is not a satisfactory explanation for the cause.

“Problems with the display of some questions” is not what happened.

“Those questions will NOT be counted toward your exam results” is a grossly incomplete solution.

So what did happen?

Well, the ABR still hasn’t offered a technical explanation. It would seem there was an issue with mammo module of the exam. If I had to guess, the larger image file sizes in this module probably exceeded a temporary throttling of the server they were hosted on and could not be transferred to all stations as the requests timed out.

But who knows? Apparently not the ABR.

The result of whatever happened is that some examinees in Chicago couldn’t start the exam. Some of them waited nervously in the holding room at the hotel room without explanation awaiting the shuttle. Others already at the center just had to sit at their desks wondering when they would be able to start. For two hours. Which of course turns the already long day into a hellishly long one with nerves racked, tummy grumbling, caffeine wearing off, etc.

Once the exam began, some test-takers had the mammo questions. Others did not. And some had them added to the end of the test mid-way through, suddenly increasing their day by another hour. In all cases, the ABR has suggested that “those questions” won’t adversely affect their scores. This presumably means that no one in Chicago will have mammo graded. But then why add it to some people’s tests and not others? Why make someone whose test-day is already two hours delayed stay another hour for questions that won’t count? How are they going to reconcile the fact that there are psychological and fatigue effects from this mistake that have nothing to do with the “display of some questions,” and that some of this could have simply been mitigated by upfront transparency?

In the grand scheme of things, given that nobody has ever conditioned the mammo section, I imagine the ABR feels confident saying that those questions not being graded will not have a meaningful impact on the grading of the examination itself. With around 103 total fails last year, one imagines only a fraction of those would even include mammo. Even the vast majority of people affected are probably nowhere near the failing mark, unfair psychological BS notwithstanding.

A follow-up email on June 14 (almost a week later) said this (emphasis mine):

The ABR sincerely regrets the problems with the administration of the Core Exam in Chicago on Thursday, June 8, 2017. We are taking this matter very seriously and are working hard to identify the sources of the problem and the impact on affected candidates.

We don’t yet have all the information needed to determine how many candidates have been affected and to what extent. Staff worked very hard over the weekend to ensure that the Core exams administered in Chicago and Tucson this week would go smoothly, and we have had no issues.

I want to emphasize that any candidate impacted by last Thursday’s difficulties with the breast imaging content will not have those items counted against their scores. We don’t expect anyone to have problems qualifying for MQSA.

How can you not know who was affected? The nature of this problem should have made it obvious who was affected during the examination itself. What they mean is that—despite getting into the business of test administration—the ABR never anticipated technical difficulties, had no meaningful system in place for troubleshooting or identifying issues, and had no contingency plans formed to deal with this eventuality.

Also missing: acknowledgment of any the issues outlined above outside of the “difficulties with the breast imaging content.”

And: you don’t “expect” problems with MQSA? The MQSA requirements only state that the radiologist be board-certified, not that the boards actually contain mammography. Of course this shouldn’t be a problem. But if you anticipate that there could be an issue, perhaps you should get some clarification before dropping a half-baked position-statement.1

 

Let’s go back to the underlying arguments for how we got here in the first place.

From the ABR FAQ:

Why do I have to go to Chicago or Tucson instead of a local testing center for diagnostic radiology exams?
With the transition to more image-rich exams with advanced item types, the ABR has built two exam centers in Chicago and Tucson to administer all diagnostic radiology exams. At this time, commercial test centers do not have the technology or means available to support these kinds of exams.

More detail from the 2014 Core Exam FAQ & misconceptions presentation:

Why can’t I just go to a PearsonVUE center to take this test?
• Modular content difficult for PV
• PV can’t handle case structure on their software
• PV monitors aren’t calibrated, can’t control lighting
• Aim: to have distributed exam. We are working on system to implement

So, now in 2017, we can firmly debunk these arguments

1. Modular Content

The content is not bizarrely or unique modular. First, this doesn’t really matter (even the very long Step exams are broken up into multiple modules). In years past, the modules for different sections were given in succession (breast, then cardiac, then GI) though lumped seamlessly into one large mega-module as you progress through the day. This year the modules were jumbled and topics jumped around. Thus there are just two days of relatively unmodular content.

2. PV can’t handle case structure on their software

This is only plausible if the ABR’s software is particularly poorly written. The USMLE also has multiple different case structure formats, including videos, images, and interactive fake physical exams, not to mention Step 3’s ludicrous choose-your-own-adventure CCS program. If we need to get rid of the two or three “drag the X” format questions per test in order to do a disseminated exam, I think we can all agree the collective radiology hivemind would acquiesce.

3. PV monitors aren’t calibrated, can’t control lighting

After this year’s difficulties, one can easily argue that there is no point having a “well-calibrated” monitor that can’t even show the carefully curated “Angoff-validated” questions in the first place. I’ll admit, the lighting is nicely dim. As a practical matter, few images are of sufficient quality for the lighting to be a plausible limiting factor. Most of the MR looks photocopied from books published in the 1980s. Residents take the ACR in-service exam in droves every year. The criticism there has always been the exam itself; not the testing software nor the ambiance of the venue.

4. Aim: to have distributed exam. We are working on system to implement

2018 sounds like a great year to start.

 

The costs of the ABR’s exam paradigm are absurd

There are almost 1200 graduating radiology residents every year (1149 took the core in 2016; 91% passed). Every class contributes $640 per person per year for a total of $3 million per graduating class over the course of a four-year residency ($4.6 million total when including the extra two years to take the Certifying Examination). That also means that the ABR rakes in around $750k per class per year and $3 million per year from residents alone. Not to mention the $340/year for every single radiologist in the MOC phase. Or the $3000+ to take subspecialty exams like neuro or VIR.

To reiterate: the class that just took this failed exam gave the ABR on the order of $3,000,000 to take this test. This figure doesn’t include the additional costs for the honor of traveling across the country to spend two days in a hotel to actually take the exam (at least another $500,000 per year).

If you can’t get photos and radio buttons working consistently on an operating budget of millions, then you’re doing it wrong.

 

Having a decent test is an important noninterpretive skill

When the ABR decided to start from scratch and write a new exclusively computer-based exam, they chose to become not just test-writers but test-administrators. No one forced the ABR to write a test that no high-volume testing center could implement. When you take over something this important, you have to do it right, and you should be completely accountable for your performance. Transparency should not be optionable. The way the Core and Certifying exams were created, graded, and handled is a poorly conceived and unnecessarily obfuscated embarrassment (e.g. why does the Certifying exam even exist?).

You don’t just say things like2

we had a mysterious technical difficulty but also we totally fixed it we promise though actually we don’t know what happened or exactly to whom it happened but also don’t worry about those questions they won’t count for anyone because for real we don’t know who had them or didn’t have them or if they had them how pretty they looked so trust us also by the way your annual fee is due.

Since noninterpretive skills are an important part of the Core exam, let’s just say that a 6% failure rate for successful Core exam administrations is a far cry from Six Sigma.3

My new book: Medical Student Loans

06.22.17 // Finance, Writing

My second book, Medical Student Loans: A Comprehensive Guide, is now out. It’s a novella-length treatment of student loans specifically for physicians and written to cover the topic for all levels: premeds, medical students, residents, and attendings. It’s especially helpful for graduating MS4s and by its nature also covers important basic financial literacy in a hopefully non-threatening way.

In other words, I hope you like it.

Despite years of writing about student loans on this site, it was a ton of work to put this together and finally get it out to the world. To celebrate, I’ve made it completely free to download from Amazon until the end of Sunday, June 25.

MSL will also be part of the Kindle Unlimited program for the next three months. You can get a 30-day free trial if you need another way to read it for free.

Consider it your first few hours of CME.

Standard Ebooks is an awesome long overdue idea:

Standard Ebooks is a volunteer driven, not-for-profit project that produces lovingly formatted, open source, and free public domain ebooks.

Ebook projects like Project Gutenberg transcribe ebooks and make them available for the widest number of reading devices. Standard Ebooks takes ebooks from sources like Project Gutenberg, formats and typesets them using a carefully designed and professional-grade style guide, lightly modernizes them, fully proofreads and corrects them, and then builds them to take advantage of state-of-the-art ereader and browser technology.

What a great project.

// 06.21.17

Explanations for the 2017-2018 Official Step 2 CK Practice Questions

06.10.17 // Medicine

The updated 2017-18 official “USMLE Step 2 CK Sample Test Questions” PDF, released in May and available archived here.

The PDF set is completely unchanged from last year. You can read the complete explanations for last year’s set here. A helpful reader Jarrett Lever made a PDF to online version conversion list.

 

As for the updated multimedia questions found only in the online version:

Block 1

7. A – Classic Moro reflex, entirely expected and normal until it disappears around age 4 months. If you have never seen a newborn before, also note that the mom is concerned about delayed milestones at two weeks of age, which is a red flag for BS: babies aren’t even smiling socially yet by two weeks.

Block 2

3. D – Pill-rolling resting tremor of Parkinson’s disease secondary to loss of dopamine neurons in the substantia nigra.

18. A – I’m going to point out that a normal healthy kid with no cardiac history or symptoms and no family history of sudden cardiac death for a pre-sports physical is probably going to have a benign exam no matter what you think you hear. HOCM is what you want to exclude theoretically, but here we don’t have a real systolic murmur, just a little vibratory flow murmur at LLSB.

33. E – This one is a bit silly. The lung exam is normal outside of the super common basilar crackles. Everything except for PE you would expect to hear a more impressive auscultation abnormality. But for this question: B and C take longer than 3 days. D we would expect fever, productive cough etc. Bronchitis would be possible, but still more often to have at least productive cough if not fever. PE, on the other hand, classically has a nonproductive cough, hypoxemia, and tachycardia. All three are present. And then they mention her med: OCPs, which are an important predisposing factor for PE in young women for whom it is otherwise a rare entity. Young lady on OCPs is a classic set-up for an STD question (who needs condoms?) or a PE question, one of the two.

Block 3

12.1 D – Statistical significance (a low p-value) does not equal clinical significance. A favorite teaching point when it comes to interpreting literature.

12.2 C –A & D are conjectures: the kind of statements people drop inappropriately in the conclusion of a weak paper to make it sound important. E is an exclusion criterion. B is the opposite: including 0 is equivalent to something not being significantly different.

Navient is still lying to borrowers despite lawsuit

06.09.17 // Finance

Unsurprisingly, Navient is still lying to borrowers despite the ongoing lawsuit (for misleading borrowers) from the Consumer Finance Protection Bureau.

I was talking to a fellow resident last week. She has almost a half million dollars in student loans from medical school and has been repaying in IBR. She recently got married, and her husband, also a resident, thankfully doesn’t have any student loans himself. Unfortunately, this also means that next time she recertifies her income, her payments are going to basically double thanks to the addition of his additional income. Once he becomes an attending, they would go up even further.

Given the possibility (and desirability) of public service loan forgiveness for her given her long training and very high loan burden, which would result in more money forgiven than she borrowed in the first place (due to negative amortization during residency), her goal should thus be to minimize payments in advance of this goal. I recommended she switched to Pay As You Earn (PAYE) to reduce her payments now and file her taxes separately from her husband next year (they weren’t married for tax purposes this year).

And here comes the lie: she called Navient for guidance, and the customer service representative told her she was ineligible for PAYE because she held a loan from 2009.

Anyone who knows anything about federal repayment programs or has the capacity to do a simple Google search would know that the limitations for PAYE eligibility have nothing to do with the year 2009. They are the following:

To qualify for the PAYE Plan you must also be a new borrower as of Oct. 1, 2007, and must have received a disbursement of a Direct Loan on or after Oct. 1, 2011. You’re a new borrower if you had no outstanding balance on a Direct Loan or FFEL Program loan when you received a Direct Loan or FFEL Program loan on or after Oct. 1, 2007.

So, nothing before 2007 and at least something after 2011. In other words, 2009 was a great vintage for PAYE. Full bodied and expensive with more than a hint of scut.

From the Consumerist:

The lawsuit alleges that for years Navient engaged in a series of illegal and deceptive practices, including providing borrowers with incorrect information, processing payments erroneously, and failing to address customers’ complaints.

Sounds familiar.

As always, it’s difficult to know if the servicer customer service reps are willfully ignorant or malicious, but they are routinely wrong (and seemingly proactively so).

My advice to anyone ever calling a servicer with a question is to already know the answer before you ask it. Find some official government verbiage to back up your interpretation. You should be looking for confirmation, not advice, and if the answer you receive isn’t the answer you’re expecting, find out exactly why. If the person you’re talking to doesn’t know, then get them to give you to somebody else.

More Open Access Journal Shaming

06.02.17 // Miscellany

There seems to be a never-ending shaming parade of “peer reviewed” open access journals that exist to extract one-time lump sum payments from desperate authors in exchange for a publishing credit and poorly formatted PDF.

“The conceptual penis as a social construct” in Skeptic Magazine takes it the next level, by also lampooning an entire discipline of academic thought. The article’s approach, summarized by the authors:

We didn’t try to make the paper coherent; instead, we stuffed it full of jargon (like “discursive” and “isomorphism”), nonsense (like arguing that hypermasculine men are both inside and outside of certain discourses at the same time), red-flag phrases (like “pre-post-patriarchal society”), lewd references to slang terms for the penis, insulting phrasing regarding men (including referring to some men who choose not to have children as being “unable to coerce a mate”), and allusions to rape (we stated that “manspreading,” a complaint levied against men for sitting with their legs spread wide, is “akin to raping the empty space around him”). After completing the paper, we read it carefully to ensure it didn’t say anything meaningful, and as neither one of us could determine what it is actually about, we deemed it a success.

This one is a fun ride.

I actually have a case report in an Open Access journal back from my I’m-going-to-be-an-interventional-radiolgoist days (it wasn’t all that good, and my colleague did most of the work, bless him). We submitted but didn’t pay—I think they may have been desperate for articles to publish and just put it up. Case Reports are almost unpublishable now outside of these types of pay-to-publish journals, which creates a bizarre counter-incentive to trying to share interesting one-offs with other physicians and scientists.

I wish—in addition to a robust mechanism for consistently sharing negative results—that there was a better mechanism for sharing research outside of peer review, which is expensive, inefficient, and, in many cases, broken.

Academic publishing is stuck in the pre-digital era. All we’ve done is move physically printed journals online behind paywalls. Comments, updates, additions? Sorry, no. Journals are static, even though science is not. It’s ripe for a big investment by a billionaire to change the status quo. Gates, Zuckerberg—you guys listening?

It’s surprising to me that people would question that obesity would have a negative effect on the brain, because it has a negative effect on so many other bodily systems,” he says, adding, “why would the brain be spared?”

– Terry Davidson, director of the Center for Behavioral Neuroscience at American University in Washington, D.C, speaking to NPR about new research concerning obesity’s effects on memory.

// 05.31.17

DeVos Doubles Down: One Servicer to Rule Them All

05.25.17 // Finance

After rolling back modest consumer protections last month, Education Secretary Betsy Devos’ tenure has released its biggest change yet.

In an example of doublespeak that would make Orwell proud, DeVos said this in the announcement:

From day one, my priority as Secretary of Education has been to put students’ needs first. This amended solicitation does just that. It maintains superior customer service and borrower protections while increasing oversight and protecting taxpayers.

…

With changes in the new amendment, we have simplified the process to ensure meaningful borrower protections while saving taxpayers more than $130 million over the next five years. Savings are expected to increase significantly over the life of the contract. Borrowers can expect to see a more user-friendly loan servicing interface, shorter email and call response times and an improved payment application method that will maximize the benefit of each payment the borrower makes. Our amendment makes no changes to repayment plan requirements.

But what the amendment actually does is consolidate the servicer program into one mega-contract in 2019 with no meaningful stipulations for accountability and no demand that the contractor work in the best interest of (or even be fair to) borrowers.4

Yes, the solution to a bad servicing situation is just to consolidate the program and give it to one private for-profit company, and the shortlist of three finalists includes Navient, universally considered the worst servicer, the one with the most complaints, the one with the most opaque and unusable website, the only one currently being sued by another government agency, and the one that despite that lawsuit continues to mislead borrowers.

Here is how Navient responded to being sued by the government for misleading borrowers:

There is no expectation that the servicer will “act in the interest of the consumer.” Courts therefore routinely hold that servicers and lenders “do not owe borrowers any specific fiduciary duties based upon their servicer/borrower relationship.”

From the Consumerist:

According to the CFPB’s April complaint snapshot, Navient was the most complained about student loan servicer, receiving an average of 1,400 complaints from Nov. 2016 to Jan. 2017, a 1,073% increase from the same three-month period the year before.

The Direct loan program tops $1 trillion. Navient already services over $300 billion in student loans and profited around $308 million from servicing those loans last year, down from $338 million in 2015 in part because of $17 million in extra legal expenses for basically being an awful company.

GreatNet, one of the other three finalists (the third is PHEAA aka FedLoan), is the combined venture of Great Lakes and Nelnet, two of the four largest servicers. Both companies have yet to be sued by the government and aren’t quite as hated, which for all I know may disqualify them from making a competitive bid in the current regime.

The budget savings of $130 million over five years is only a small fraction of total servicing expenses, but—more importantly—the increased cost of the Obama-era plan was to pay for genuine customer service to help reduce delinquency and default. Reduced default would likely generate more payments from borrowers, thus paying for some if not all the difference (Americans currently hold $137 billion in defaulted federal loans).

Lastly, the government profits from student loans. This is not a situation in which a budget shortfall affects uninvolved taxpayers; it’s a scenario in which a portion of the profits from student loans are reinvested into the same system to help American citizens who utilize it.

Monuments and Symbols

05.23.17 // Miscellany

New Orleans Mayor Mitch Landrieu on the removal of four historic statues commemorating the confederacy:

These statues are not just stone and metal. They are not just innocent remembrances of a benign history. These monuments purposefully celebrate a fictional, sanitized Confederacy; ignoring the death, ignoring the enslavement, and the terror that it actually stood for.

After the Civil War, these statues were a part of that terrorism as much as a burning cross on someone’s lawn; they were erected purposefully to send a strong message to all who walked in their shadows about who was still in charge in this city.

…

Another friend asked me to consider these four monuments from the perspective of an African American mother or father trying to explain to their fifth-grade daughter who Robert E. Lee is and why he stands atop of our beautiful city. Can you do it?

Can you look into that young girl’s eyes and convince her that Robert E. Lee is there to encourage her? Do you think she will feel inspired and hopeful by that story? Do these monuments help her see a future with limitless potential? Have you ever thought that if her potential is limited, yours and mine are too?

That last line.

Servicers other than FedLoan don’t handle PSLF

05.04.17 // Finance

I’ve heard a few stories of attendings calling their servicers after making years and years of payments to get started filing for PSLF and being laughed off the phone because of their high salaries.

To be clear, PSLF has no maximum salary.

The master promissory note you signed says this:

A Public Service Loan Forgiveness (PSLF) program is also available. Under this program, we will forgive the remaining balance due on your eligible Direct Loan Program loans after you have made 120 payments on those loans (after October 1, 2007) under certain repayment plans while you are employed full-time in certain public service jobs. The required 120 payments do not have to be consecutive. Qualifying repayment plans include the REPAYE Plan, the PAYE Plan, the IBR Plan, the ICR Plan, and the Standard Repayment Plan with a 10-year repayment period.

PSLF is all about months of qualifying payments made for qualifying loans while working full-time at a qualified employer. As of now (and probably forever for old/current borrowers), there is no “means test” to see if you still deserve to have your loans forgiven even though you’re a “rich doctor.”

Furthermore, the only servicer that handles PSLF is FedLoan. When you submit your first employment verification form (which the Feds recommend doing annually), you’ll be switched to FedLoan if you weren’t already with them by chance.

The other servicers—and probably especially Navient (currently being sued by the federal government for defrauding and misleading borrowers)—have a vested interest in keeping you on the rolls so that they can continue to make money off your payments. When you call the servicer, you’re getting some random employee who is probably making around twelve bucks an hour whose primary role is to provide customer service and troubleshooting with the website, not provide good financial advice. They are much much more likely to deal with somebody on the phone trying to get out of delinquency or default than they are to talk to somebody who is approaching 10 years of payments and is ready for public-service loan forgiveness. Despite the government stating that customer service is a priority, the servicers essentially have no fiduciary duty to work in your best interest.

Since the first loans won’t be forgiven until October 2017, no one can guarantee that there won’t be attempts to limit the damage from the somehow-unexpected popularity of this program, but that hasn’t happened yet and is likely to take some time to occur. Do not take your servicer seriously on this if you think you should otherwise qualify. Just get the paperwork filed out and submit it. Worst thing that happens? Your servicer is changed and you have to set up autopay again.