Ben White

Tomorrow is a new day

11.11.20 // Reading

Finish every day and be done with it. You have done what you could; some blunders and absurdities no doubt crept in; forget them as soon as you can. Tomorrow is a new day; you shall begin it well and serenely, and with too high a spirit to be cumbered with your old nonsense.

—Ralph Waldo Emerson, in an 1854 letter to his daughter.

It’s my birthday today, and this is something I think about when considering my relentlessly increasing age.

The best day for a new positive change is always today. But we’re very fallible human beings and I’ve been stress eating for about six months, so barring that, let’s not discount tomorrow either.

“The world needs you at the party starting real conversations, saying, ‘I don’t know,’ and being kind.”

—Charlie Kaufman

// 11.04.20

How the ABR’s MOC OLA scoring works

10.31.20 // Radiology

I’ve been talking to the ABR for a bit recently about OLA scoring. I’ve made no secret about my opinions on the actual content of the ABR’s 52-question-per-year MOC qbank and the likelihood of its ability to ensure anything meaningful about a radiologist’s ability to practice.

But I’ve been especially interested in how OLA is scored. For quite a while there was a promise that there would be a performance dashboard “soon” that would help candidates figure out where they stood prior to the big 200-question official “performance evaluation.”

And now it’s here. And, to their credit, the ABR released this blog post–How OLA Passing Standards and Cumulative Scores are Established–that actually really clarifies the process.

How Question Rating Works

In short, the passing threshold is entirely based on the ratings from diplomate test-takers, who can voluntarily rate a question by answering this: “Would a minimally competent radiologist with training in this area know the answer to this question?”

I think that wording is actually a little confusing. But, regardless, the fraction of people who answer yes determines the Angoff rating for that question. The passing threshold is above that percentage and failure is below, averaged out over all the questions you’ve taken. Because different people get different subsets of a large question database, currently “the range of passing standards is 50% to 83%.”

That’s a big range.

One interesting bit is that a question isn’t scorable until it’s been answered by at least 50 people and rated by 10, which means that performance can presumably change for some time even when not actively answering questions (i.e. if some already answered questions are not yet scorable when first taken). Back in the summer of this year, approximately 36% of participants had rated at least one question. It may take some time for rarer specialities like peds to have enough scorable questions to get a preliminary report.

The Performance Report

The current performance dashboard is a little confusing. Here’s mine:

It includes neither the actual passing threshold nor your actual performance. It’s got a cute little graph (with currently only one timepoint), but it reports performance simply as a percentage above or below the passing standard. I wasn’t quite sure what that meant mathematically, but the ABR actually addressed that in the post:

If the participant is meeting or exceeding that passing standard, their current cumulative score will indicate what percentage above the passing standard has been achieved. For example, if the average rating of the questions is 70 percent, the participant is required to answer at least 70 percent of the questions correctly within the set to meet the passing standard. If a participant’s current cumulative score indicates that they are 21 percent above the passing standard, then the participant would have successfully answered 85 percent of their questions correctly.

Based on that paragraph, their math here for passing individuals:

your percentage correct = passing threshold*percentage above + passing threshold = passing threshold*(1+percentage above)

In their example, that’s 85 = 70*.21 + 70 = 70*1.21

I’d rather just see the underlying numbers because the ABR doesn’t make it easy to see your cumulative performance at a glance. It’d be nice to see the actual passing threshold and your actual percent. But with this information, you can now–with some effort–estimate these numbers.

First, get your own raw performance. This isn’t straightforward. You’ll need to go to “My OLA History.” Filter to count your incorrect questions by clicking on “Your Answer” and selecting “incorrect.” Manually count that hopefully small number of questions and use that versus the total number you’ve answered (if you’ve been doing more than 52 per year, it’s going to be tedious to count manually).

So, for example, I’ve answered 104 questions and gotten 3 incorrect, which nets me a performance of 97%. My percentage above passing is reported as 50%, though it’s worth noting that the scale maxes out at plus/minus 50, which mathematically can’t be correct if the minimum passing threshold can be as low as 50%, so who knows. Anyway:

97 = X*1.5

X = 97/1.5 = 64.667

So we can estimate that I needed to get ~65% correct to meet the passing threshold given my question exposure, which is right in the middle of the current reported range.

Caveats include the fact that you don’t know how many if any of your questions are currently non-scoreable nor is it easy to see how many if any questions were subsequently nulled due to being deemed non-valid, so this really is just an estimate.

In related news, it’s lame there isn’t a way to see exactly how many questions you’ve answered and your raw performance at a glance. I assume they are trying to simplify the display to account for the varying difficulty of the questions, but it feels like obscuration.

Self-governance

The ABR has stated that the OLA question ratings from the community are all that’s used for scoring. While volunteers, trustees, etc are all enrolled in OLA and presumably rating questions, the ABR says what the greater community says is what goes.

So we are to judge what a radiologist should know based on what about 1/3 of MOC radiologists think a radiologist should know. There is no objective standard or aspirational goals. Radiologists can cherry-pick their highest-performing subspecialties and maintain their certification by answering a handful of questions in their favored subjects that are essentially vetted by their peers, who may not exactly be unbiased in the scoring process.

I don’t actually necessarily see a problem with that.

What I do see is more evidence of a bizarre double standard between what we expect from experienced board-certified radiologists and what we expect from trainees.

In court, the ABR has argued that initial certification and maintenance of certification are two aspects of the same thing and not two different products. As someone who has done both recently, they don’t feel the same.

Unlike OLA, neither residents nor the same broad radiology community has any say in what questions meet the minimal competence threshold for the Core and Certifying exams. For those exams, small committees decide what a radiologist must know.

You earn that initial certification and suddenly all of that mission-critical physics isn’t important. Crucial non-interpretive skills are presumed to last forever. And don’t forget that everyone should know how to interpret super niche exams like cardiac MRI (well, at least for two days).

It either matters or it doesn’t.

Why does physics, as tested by the ABR (aka not in real life), matter so much that it can single-handedly cause a Core exam failure for someone still in the cocoon of a training program but then suddenly not matter a lick when the buck stops with you?

The fact is that most practicing radiologists would not pass the Core Exam without spending a significant amount of time re-memorizing physics and nuclear medicine microdetails, and the new paradigm of longitudinal assessment doesn’t allow for once-per-decade cramming necessary to pass a broad recertification exam.

Meanwhile, diplomates are granted multiple question skips, which while designed to give MOC participants the ability to pass on questions that don’t mesh with their clinical practice can nonetheless be used to bypass questions that you don’t know the answer to. Especially for the performance-conscious and those with borderline scores, this creates a perverse incentive to miss out on rare learning opportunities.

But my point here is not that OLA should be harder.

My point is that certification should mean the same thing with the same standard no matter how long you’ve held it.

 

 

 

Bigger isn’t always better

10.19.20 // Reading

From Company of One: Why Staying Small Is the Next Big Thing for Business by Paul Jarvis:

I like reading things that are not about medicine and seeing what lessons cross domains and apply. The premise of Jarvis’ book is that growth for growth’s sake is a terrible business model for many businesses and especially the many people that run them. It’s okay to be big, but any decision to grow the enterprise should be an active choice considering all factors and not just the default MO taken from the recent start-up culture of silicon valley.

My wife left her employed university position last year to start a solo private practice, and it’s been wonderful. It’s not hard to see how there are so many downsides to being part of large company or expanding your own business too much. On the former, no control. On the latter, so much time managing the machine that it’s easy to completely lose track of why you started the practice and what you liked about it.

But here are four passages that also resonated with me as a physician and educator:

Miles Kington, a British journalist, reportedly said that “knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad.” We should never assume that having an abundance of knowledge is the same as having an abundance of wisdom.

This is the problem with equating performance on a knowledge assessment like a high-stakes multiple-choice examination with real-life competence. Knowledge is important, but it doesn’t mean you can perform in your field.

More isn’t better—better is better. There are advantages to putting in the time and effort to master a skill, but there’s also a great need for balance.

“More isn’t better” is a real truism for so many things in our world. We should question whenever something or someone simply wants more of us: more hours, more years of training, more free labor, more notches on the CV belt. It’s not necessarily that the “more” is inherently bad–it almost never is–but that doesn’t mean it’s worth it.

Sukhvinder Obhi, a neuroscientist at McMaster University, coined the term “power paradox” to describe what happens when we gain power through leadership: we subsequently lose some of the capabilities we needed to gain it in the first place—such as empathy, self-awareness, transparency, and gratitude.

The power paradox explains much of the bullying we see within strict hierarchies: how excrement rolls downhill from the top of a poorly-run organization all the way to the youngest least experienced students who are just doing their impressionable best to insulate themselves from the worst of indoctrination as they grow.

[Cal Newport, author of So Good They Can’t Ignore You] believes that we need to be craftspeople, focused on getting better and better at how we use our skills, in order to be valuable to our company and its customers. The craftsperson mind-set keeps you focused on what you can offer the world; the passion mind-set focuses instead on what the world can offer you.

There are some people who in life (or medical school), are confident they know exactly what they want. They are passionate about dermatology and orthopedic surgery. That’s great.

But I don’t think there’s anything wrong with the rest of the world, those who simply don’t know or seem to be missing that “passion.” I agree with Newport that passion is something you can grow through competence and the craftsmen mentality. There are no perfect jobs or fields. There are good and bad aspects to everything, and suggesting otherwise drives so much anxiety in the specialty and residency-selection process.

Success has more to do with you, your goals, and your perspective than it does with exactly what box you place yourself in.

 

What Money Buys

10.14.20 // Finance, Reading

From How to Think About Money by Jonathan Clements:

First and foremost, money buys time and autonomy. Secondarily, it buys experiences. Last, and least, it buys stuff, and more often than not, the stuff we buy makes us miserable.

Most people live their lives with these in the opposite order, but Clements is absolutely right.

You don’t have to be a FIRE-fanatic to realize that setting up your professional life, spending, and saving to optimize for number one is the winning strategy.

Simple Sabotage

10.12.20 // Miscellany, Reading

Simple Sabotage: A Modern Field Manual for Detecting and Rooting Out Everyday Behaviors That Undermine Your Workplace is a book inspired by a real World War II CIA field manual called “Simple Sabotage” that was written to help “guide ordinary citizens, who may not have agreed with their country’s wartime policies towards the US, to destabilize their governments by taking disruptive action.” You can read the declassified original document at that link.

It’s short and fascinating and much of it is timeless. Operationally, it functions as a “how-not-to” for creating an efficient organization. The CIA’s top 3 takeaways:

1. Managers and Supervisors: To lower morale and production, be pleasant to inefficient workers; give them undeserved promotions. Discriminate against efficient workers; complain unjustly about their work.

2. Employees: Work slowly. Think of ways to increase the number of movements needed to do your job: use a light hammer instead of a heavy one; try to make a small wrench do instead of a big one.

3. Organizations and Conferences: When possible, refer all matters to committees, for “further study and consideration.” Attempt to make the committees as large and bureaucratic as possible. Hold conferences when there is more critical work to be done.

These points were once felt to be a great way to sabotage Nazi Germany, but they seem to have been voluntarily taken up by most modern American businesses.

A good example from the book is the “obedient saboteur,” someone who–by doing exactly what he’s told to do–is actually making things worse:

This problem can be particularly acute in organizations with a culture of “continuous improvement.” Continuous improvement is a business philosophy created by W. Edwards Deming in the mid-twentieth century. This philosophy thinks of processes as systems and holds that if each component of the system constantly tries to both increase quality and reduce costs, efficiency and success will follow. But taken to an extreme, even continuous improvement can lead to sabotage.

One company we know had a call center manager driving his team to move from an average pickup speed of 1.4 rings to 1.2 rings. The division head asked, “How often do callers abandon us after only 1.4 rings?” “Almost never,” he was told. “Virtually no callers who actually intended to call us hang up before the third ring.” Yet the call center manager persisted in trying to ensure that all calls were answered more quickly each year. Why? Because getting the phones answered quickly was his job—by definition, quicker was better. He never thought to question whether he had crossed the threshold where process had overridden outcome. He had become one of the Obedient Saboteurs. If you asked him why he was trying to lower pickup times, he would tell you that faster pickup means improved customer experience. That’s true—but the threshold is three rings. Once you get below three rings, faster pickup times don’t continue to improve the customer’s experience anymore.

The problem we see, time after time, is that nobody bothers to go back and tell the call center managers of the world to go continuously improve something else.

To keep this kind of sabotage out of your group, step back and conduct a formal review of any continuous improvement programs you have in place. If they aren’t relevant anymore, pull the plug.

Also see: measure what matters.

To fight back, ask yourself:

What is the stupidest rule or process we have around here?

What are the three biggest obstacles you face in doing your job?

If you could rewrite or change one process or procedure, what would it be and why?

A lot of quality improvement isn’t real:

It’s adding clutter. It’s replacing content with process.

We should be just as ruthless when evaluating quality measures and metrics as we are with the fail points that inspire them.

Studying during residency

10.09.20 // Medicine

Here are some questions I received a long time ago about studying during residency:

  1. Do you have any thoughts on studying to become a better doctor?
  2. What and how do you study when not preparing for some fun standardized test?

The easy answer for the latter is that in our modern system of medical education and board certification, you’re always preparing for a fun standardized test.

But I think the real answer to both of these questions to make it about your patients as much as possible.

Approach

You should always consider a broad differential and use real patients as opportunities to consider and learn about alternative diagnoses. If you have the motivation, consider further broadening your differential or treatment considerations unnecessarily just to have a relevant excuse to learn more about given topics. Think “I know it’s not disease X, but what if it was?”

Grounding as much learning as you can with patient care will give you the broad foundation you need in your field to build on when new things come along. But on top of that, linking up the information you learn from resources and articles with real live patient memories gives that information more staying power and helps you fight against the forgetting curve.

In medical school, you are hyper-directed in the content you must master, though in many cases toward low-yield material and wasted energy. In residency, you have more leeway toward becoming an expert in things that will directly impact your practice.

Content

I think for many residents it’s generally difficult to “study” in the medical school sense of systematically sitting down with a book or resource without a test looming in the near future. You’ll have the in-service, which you can use as motivation for dedicated review and MCQ fun, but preparing for your patients/daily activities is something you can do continually and, when done aggressively, can cover a large fraction of the relevant material. You may find relevant book chapters helpful on occasion, but squeezing in UptoDate articles and occasionally reading their references, the infrequent Google Scholar/PubMed search, and reading up on the next day’s procedures/surgeries (or the equivalent for your field), are going to work well in general.

I do think that Anki style flashcards and question banks are still good tools. If you have rotations that contain relatively well-defined material, these may even be straightforward to consider and implement on a schedule. In radiology, for example, it’s pretty easy to (at least plan) to read a book on chest radiography and do the chest RadPrimer MCQs on a dedicated chest rotation.

You may need to give yourself a long-term curriculum to work through, whether that’s guided by a commercial question bank or just following the table of contents of a gold standard textbook.

The Crux

The real limitation here is time and energy. Residency is busy. Call shifts can be brutal, and by the time you recover, you’re on call again. You may have a spouse who needs support and children who deserve a parent. And people keep trying to dump boring research projects on you. Sometimes, something’s gotta give.

But what I would say is that you’ll be more able to learn efficiently if your outside-work-life is harmonious enough that you can be fully present for your daily work. That part is almost non-negotiable. So before you guilt yourself for not studying enough at home, make sure you’re doing the things that you need to in order to recharge your battery to be a thoughtful physician for your patients. The trite lines with all the burnout talk out there is that you need exercise, eat healthily, and spend time nurturing your meaningful relationships. And you know what? That’s probably a good start.

Most of the things that really have an impact will come up as you engage actively with patient care, but some of the other BS will only come when its time to review for that next standardized high-stakes exam.

Ultimately, you caring about your patients as individual human beings and paying attention are the two most important things you can do to provide good care and to learn.

Status Quo Bias

10.07.20 // Miscellany, Reading

From Machine, Platform, Crowd: Harnessing Our Digital Future by Andrew McAfee and Erik Brynjolfsson:

Research in many different fields points to the same conclusion: it’s exactly because incumbents are so proficient, knowledgeable, and caught up in the status quo that they are unable to see what’s coming, and the unrealized potential and likely evolution of the new technology.

This phenomenon has been described as the “curse of knowledge” and “status quo bias,” and it can affect even successful and well-managed companies.

There are a lot of bad actors in healthcare that I would love to see fall prey to the curse of knowledge.

When “value” became shorthand for “economic worth”

10.05.20 // Reading

From Doughnut Economics: Seven Ways to Think Like a 21st-Century Economist by Kate Raworth (emphasis mine):

Despite such misgivings from the twentieth century’s two most influential economists, the dominance of the economist’s perspective on the world has only spread, even into the language of public life. In hospitals and clinics worldwide, patients and doctors have been recast as customers and service-providers.

There may be no perfect frame waiting to be found, but, argues the cognitive linguist George Lakoff, it is absolutely essential to have a compelling alternative frame if the old one is ever to be debunked. Simply rebutting the dominant frame will, ironically, only serve to reinforce it. And without an alternative to offer, there is little chance of entering, let alone winning, the battle of ideas.

But when political economy was split up into political philosophy and economic science in the late nineteenth century, it opened up what the philosopher Michael Sandel has called a ‘moral vacancy’ at the heart of public policymaking. Today economists and politicians debate with confident ease in the name of economic efficiency, productivity and growth—as if those values were self-explanatory—while hesitating to speak of justice, fairness and rights. Talking about values and goals is a lost art waiting to be revived.

I love that.

And the example of healthcare I think is exactly right. Everything is a business–that’s unavoidable. It isn’t even a bad thing. Lose money and you won’t be in business very long. But not all businesses need to be organized with the primary purpose of optimizing productivity and growth.

Good patient care is inefficient. Talking to people–understanding their perspective and helping them become active participants in their health–takes time. A patient visit was never meant to be an assembly-line 15-minute med check.

It’s not that we should applaud or celebrate inefficiency. There is plenty of waste to trim in any enterprise. It’s that these ideas–efficiency, productivity, and growth–should be tools to achieve meaningful ends, not the primary endpoint. Measure what matters.

And if some of that extra “value” makes it to the actual workers? Much of our economy is predicated on individuals misallocating their income away from savings and away from optimizing their time:

As economist Tim Jackson deftly put it, we are ‘persuaded to spend money we don’t have on things we don’t need to make impressions that won’t last on people we don’t care about’.