Ben White

Recalls and Exam Security

08.20.20 // Medicine, Radiology

Out of all the reasons organizations like (but not limited to) the ABR have used as an excuse to shy away from remote content or historically relied on commercial testing centers, I strongly suspect exam security is the only one that actually matters.

While an individual not cheating on the exam is important for exam integrity, that type of exam security is a relatively straightforward n=1 problem. The real exam security that matters is the security of intellectual property. Nevermind that all these medical organizations use questions largely written and initially vetted by volunteers, losing hundreds of proprietary questions in one fell swoop to some industrious malcontent is the real fear.

The recent utter failure of the American Board of Surgery’s virtual testing also makes the point: once people have seen a significant fraction of the questions for a high stakes exam, it’s back to the drawing board. A doomed effort to administer an exam remotely sets an organization back by months, at the least.

That’s because recalls—people sharing memorized exam content—are a big deal. In fact, news about the universal use of recalled study questions for the radiology oral boards back in 2012 was the driving force behind the creation of the MCQ-only Core and Certifying exams, administered only on-site in bespoke testing centers created by the ABR itself in Chicago and Tuscon that sit gathering dust for most of the year.

While recalls are against organizational policy (and thus certainly something an individual should not do), the focus on recalls as a destabilizing force for the fairness of exams is…lame. Literally every important high-stakes exam including the SAT, MCAT, USMLE, and the various board exams have engendered a massive test prep industry around offering nearly the same thing: questions written—sometimes by the same people!—to exactly simulate those very same exams. Many, including most of the USMLE prep products, use software that even completely mimics the test software down to the pixel.

But I want to posit a fundamental misunderstanding:

A Really Meaningful Modern Test Shouldn’t Rely on Hiding its Content

Imagine a radiology exam had a question demonstrating a benign hepatic hemangioma on CT or MRI. Imagine a second-order question asking about management (nothing). If that information were put into a series of recalls, it would be meaningless. Every radiology resident should get the question correct because it is relevant to radiology practice and unavoidable during normal training. And if the exam, like the USMLE licensing exams or the various specialty board exams, purports to be a measure of minimal competency for safe independent practice, then all the questions should be relevant to daily practice.

If someone has mastered all of the relevant “recalls,” then they would presumably be ready to practice radiology. It’s okay if the fraction of answered questions is high if we expect the questions to reflect things we really want everyone to know. Mastering all of this exam content is exactly what we want trainees to do!

Recalls only matter in two situations:

  1. If exams need to include questions designed to differentiate high-level performance, separating the best from the average. In this setting, questions that should be really challenging become easy, throwing off the balance of exam difficulty. It’s precisely because the designers want to give you brainbusters that the element of surprise is key. But in a world where high-quality informational content is at your fingertips, this component of mastery is increasingly irrelevant. The things we really want people to know quickly, like how to manage a true emergency like a code or how to drop a line, are never satisfactorily going to be assessed with a multiple-choice question. That’s what a real-life training program is for. Performance on an MCQ test is typically only good at predicting future performance on other MCQ tests.
  2. Questions that really really test critical thinking. In this setting, the examinee shortcuts the thought process and arrives directly at the answer, defeating the purpose of the question. While the MCAT contains a fair number of reasoning questions along these lines, this is rarely the case in real life for medical licensing and board exams.

Every question bank for every major test is composed of glorified recalls. Pretending otherwise is silly. If all questions contained important material and the ability to answer them reflected meaningful knowledge and competence, then someone able to memorize the plethora of recalled questions would be exactly what you’d want: qualified.

Lessons from the American Board of Surgery’s Virtual Board Exam Debacle

08.18.20 // Medicine

Lesson 1: Beware Third Party Services

During the slow drama that has been the delivery of high-stakes during the pandemic, some organizations like the American Board of Radiology resisted agreeing to a virtual exam for so long precisely because they couldn’t control the environment, delivery, and security. Unfortunately, we have some great evidence that offloading exam security to a third-party company can easily fail on those metrics as well.

Just ask the American Board of Surgery, which hired Verificient’s Proctortrack, an “industry leading Online Proctoring / Remote Invigilation solution.” Leaving aside how creepy “invigilation solution” sounds, Proctortrack has a 1-star rating on the app store for being generally awful. And despite the platform’s buzzwords of AI, computer vision, and machine learning, there were both serious mechanical and human failings.

The ABS went from a single day 8-hour exam to a two-day 4-hour exam, just to make sure the platform could handle it. It couldn’t.

ProtorTrack is extremely invasive. It requires download and installation of monitoring software that gives serious access over both your computer and your smartphone. The service also requires a room scan of 360 degrees to clear your workspace of nefarious intent (that last part actually makes sense to me).

So what happened?

Near instantaneous failure. The Room Scan feature failed early, and then examinees were basically left out to dry with poor phone and online customer service. People kept trying to log in and ended up banging their heads against the wall. I’ve heard an estimate that around half of the residents were able to take at least a portion of the test. But the circumstances were less than ideal, with many residents apparently reporting being harassed to complete distraction by the “live” overseas proctors.

Shortly after, the reports of identity theft, unauthorized credit card charges, and inappropriate social media contact began.

Because the test questions were exposed to a fraction of the examinees, those questions are no longer usable. The ABMS boards talk a lot about “their” intellectual property, and while these questions are written by volunteers, we shouldn’t demean the serious effort that goes into crafting and then vetting them. MCQ tests may not be a great measure of medical competence, but that doesn’t mean a lot doesn’t go into making one. Having to throw out a test is a serious setback.

I don’t know what goes into how these organizations make these sorts of operational decisions, but I don’t think it takes a clairvoyant to predict that ProctorTrack might have been a suboptimal choice.

While it might look good to an uninvolved observer, I don’t think a third-party off-site unaccountable organization is the right move for a high-stakes virtual exam. While at home exams require more effort, a residency-based in-person exam with live or focused virtual proctoring would be relatively straightforward. Regardless, video proctoring and screen recording don’t even need to be live, so long as all information is saved and can be reviewed.

It’s also hard to imagine a doctor throwing away their career just to copy some exam questions, which is really what we’re worried about here more than someone cheating to pass.

Lesson 2: Be Flexible

The ABS essentially forced residents to take the test (before they canceled it and told them they couldn’t). Taking a pass this year would use up an attempt and looks bad for programs.

One one hand, that doesn’t sound so bad if it all worked out. But it didn’t. And when things don’t work out, people’s lives are way harder. Dedicated study time, job starts, family planning—everything. Doctors build their lives and programs plan their schedules around these exams. When we live in tough times, it’s best to err on the side of accommodation.

Any board needs to have contingency plans in place that will accommodate a rapid administration to waiting candidates. Write more questions now and have them ready to deploy. Running out of questions after a failed attempt and needing to push things back indefinitely in order to write more is a terrible plan.

Lesson 3: Prioritize Communication

Announcements about the exam were done primarily through Twitter. Emails and website updates came universally after a multi-hour delay. Twitter is fine, but Twitter shouldn’t be used as a replacement for individualized communication. When it comes to a career-defining exam, people should be getting so many emails (or potentially opt-in text messages) that they almost find it annoying.

By the time the ABS finally published a FAQ on its website, days had passed.

When the Resident Association of the American College of Surgeons (RASACS) and ABS hosted a virtual town hall, it was announced within five hours of the event and then limited to in-advance sign-ups of only 400 people, a small fraction of the affected residents.

Take-Home

I don’t want to suggest that doing a virtual exam for the first time in history is a straightforward endeavor.

What I do think is clear is that an organization whose sole purpose is to create and administer tests needs to be nearly flawless on that execution. Offloading tasks to third parties, even the commercial testing centers so commonly used, has long resulted in a suboptimal and often degrading experience. Third-party proctoring is apparently no different. I appreciate that ABMS member boards are not tech companies and will always use contractors to accomplish a variety of tasks, but that does not absolve them of responsibility for the final product.

Lastly, communication and accountability shouldn’t be an afterthought. Just because residents have no power over their respective boards doesn’t mean they aren’t worthy of consideration and respect.

Buying Disability Insurance As a Medical Student

08.12.20 // Finance, Medicine

Let’s start by saying that I’m certainly not the only person on the internet that thinks it’s critical for all doctors to buy true own-occupation disability insurance that protects you in the event that you become disabled and can’t earn your full income. Your earning potential is too high and school is too expensive to not protect. There’s a good chance you’ve already (or will soon) hear about DI all the time because your eyeballs and attention are valuable and disability insurance agents sponsor a lot of podcasts and a bunch of blogs.

And that’s actually basically okay because that helps pay the bills for a lot of meaningful content out there; these folks are paid by the insurance companies (not you), and you need to use one to buy a policy anyway.

I bought my disability insurance policy toward the end of residency, but when I look back at my post-call exhausted driving, occupational hazards (e.g. sharps), and health scares, I feel fortunate to still be healthy and to have gone through the process unscathed. Though we haven’t needed to use it yet, it’s not hard to imagine a scenario where things didn’t pan out so well or where I or my wife developed a condition that prevented us from buying insurance in the future.

I have no doubt that the best answer to the question of “when to buy disability insurance?” is as soon as feasible. And I wondered more about the logistics of buying a physician policy as a medical student, something that no one really talks about.

So I asked Matt Wiggins from Pattern to talk with me and fill in the gaps. He made a video for you, dear reader (good for anyone but especially medical students and residents), and I’ve written a post breaking down how that works in medical school and detailing my thoughts. This isn’t a sponsored post (we don’t do those around here), but I do have a relationship with Pattern if you end up checking them out to get policy quotes. (It never costs anything to see your options; agents get paid a commission if you buy a policy.)

What is disability insurance?

A disability insurance policy will provide you with monthly paychecks if you become unable to carry out the duties of your job due to disability. The more you earn, the bigger a policy benefit you can purchase (and the more it costs in premiums). Disability insurance ensures that you are protected financially when things go wrong and you can’t work the way you used to.

A good policy for a doctor is called an own-occupation policy because it pays the full benefit if you can’t do the same doctoring job you were doing as a physician when you become disabled, even if you can be gainfully employed otherwise. If a surgeon hurts her hands and can’t operate, then she’s fully disabled even if she goes on to make even more money as a consultant or another kind of doctor. The problem with most policies bundled with your employment is 1) they don’t follow you when you leave your job and 2) they typically don’t have as strong a definition as own-occupation or what defines a disability. The practical matter is that a group policy just may not cover you in real life. The sorts of policies residents have while in training are notorious for letting you down when you need them most.

When you buy a policy, you will also choose from a variety of “riders,” which are essentially optional add-ons you can purchase a la carte. Each one makes your policy more expensive but also makes it more flexible. A common example would be a “future benefit increase” rider, which allows you to upgrade to a bigger policy in the future when your income rises without needing to go through medical underwriting. Even if you develop a medical condition that is sure to result in a disability in the future, the company still has to let you exercise the rider.

When is the right time to purchase disability insurance?

So you definitely need it. The question is just the timing. Ultimately, since you can’t predict the future, the real answer is as soon as you are eligible to buy the right kind of policy and can afford it. The first part has a real answer, the affordability part is a little fuzzier.

There are two long-term financial benefits to grabbing a policy early:

—Cheaper rates based on Age and Health (you’re never younger or probably healthier than you are right now)

—Discounts from University or Training-program affiliation (range anywhere from 10-40% off the premium and will typically last the life of your policy even after you leave)

For many, a good solution is to get a very small policy towards the end of medical school. You can lock in $1,000/month in coverage for $20-$40/month, which would at least provide $1,000/month tax-free until you are retirement age should a disability happen to you in medical school. But the main benefit is that this small purchase would lock you into the ability to increase your coverage to much higher levels without the insurance company ever checking on your health again, which can be the difference between being future-proof or not. It’s all about who you are when you buy, not who you become.

How Things Differ for a Medical Student

A resident or attending buying disability insurance is able to buy an own-occupation specialty-specific policy. This is the kind that protects exactly what you do. A pre-match medical student will instead get a generic (internal medicine) policy. If you buy a policy after matching, the rate will be adjusted for the risk category (and procedures etc) of the field they’ve chosen. So an anesthesiologist, a higher-risk specialty, will have a higher rate than a family doc.

But there is some nuance. I asked Matt how that works if you specialize later on, and this is what he said:

When a doctor files an own-occupation claim, the insurance company looks to find out exactly what duties or procedures the doctor is doing at the time of disability, and that is the occupation that is protected. In other words, if a doctor buys a policy in med school and then goes on to an internal medicine residency, followed by a cardiology residency, followed by an interventional cardiology fellowship, they will be protected for the procedures they are doing as an interventional cardiologist even though they bought their policy well before they were an interventional cardiologist and their rates are cheaper than if they had purchased the policy later as an attending IC doc.

Interesting (and not what I would have expected at all).

So, if you’ve chosen a higher-risk field like a surgical specialty or anesthesiology, then you’ll also save money in the long term by getting the rates of a less risky “generic doctor” profession upfront.

Most companies will only offer policies to fourth-year students, who can purchase a benefit of up to $2500/month (which would cost in the neighborhood of $60-$100/month in premiums for men and $75-$125/month for women). Once you match, you’d be eligible to increase the resident benefit (which is 5,000/month across the board). But you don’t have to buy a policy at the maximum you’re eligible for; you can buy one that you’re confident you can afford.

Parting Thoughts

The idea that I could have skipped a couple of burritos and a latte and locked in disability insurance as a medical student is crazy to me. I waited until late in residency when we had more cash flow, but that was I think the wrong approach. I should have purchased a small policy as soon as possible. Even if I couldn’t afford the higher premiums to increase the benefit until later in training, at least I could have guaranteed that flexibility to do so upfront by being more proactive.

You should always comparison shop for DI with independent agents. In addition to Pattern, consider checking out LeverageRx as an additional source. It’s always free to get quotes (because agents are paid by commission from the insurance company). Both are affiliates, so talking to them is an easy way to support my writing.

 

 

The Cost, Price, and Debt of Medical Education

08.05.20 // Medicine

From “The Cost, Price, and Debt of Medical Education” in NEJM.

In the 1960s, 4 years of U.S. medical education could be purchased for about $40,000 (in 2018 dollars). By 2018, the average price had increased by 750%, to about $300,000; approximately 75% of students took on loans, and their average debt at graduation was $200,000. In contrast, U.S. college tuition increased by about 250% over the same period.

…between 2010 and 2018, the percentage of medical students graduating with no debt also increased — a happy result if it arose from lower prices or more scholarships, but its occurrence in the absence of those conditions suggests that medicine is increasingly a profession accessible only to the rich.

What a massive increase.

This last part, borne out by the data, is a serious problem. And a few free medical schools and a handful of scholarships aren’t going to reverse the serious financial headwinds discouraging the nonprivileged.

Of course, loan-repayment programs offer nothing to students who don’t finance their education with debt to begin with. But a more fundamental limitation is that because these programs target debt and not price or cost, they risk exacerbating high education prices for all. In theory, students who expect their loans to be fully or partially forgiven become less sensitive to the price of medical education and the price differences among schools. Anticipated debt relief — even partial or uncertain relief — reduces the already weak incentives for medical schools to compete on price and so effectively transfers money to the schools with higher prices.

Price insensitivity is a huge issue across higher education. The investment in a nebulous professional future paid with borrowed funds means that many students just accept the sticker price of whatever their dream is and only consider the consequences after the fact. For medical school, many students are just happy to be accepted. The cost differential amongst choices often only plays a role for those choosing between acceptances from lower-price home state institutions and higher-price private ones.

People often ask if PSLF will be canceled as Trump has proposed or if the government will start forgiving large swaths of student loans as many candidates proposed during the recent Democratic primaries, but that’s missing a key part of the story. Any changes to the details of loan repayment in the absence of an overhaul of higher education funding will not solve the systemic problem.

The cost of medical school—as in the amount of money it takes to train a medical student—is surely high. But is it really 3 times that of a college? Do medical students, in some situations, not also provide value to help mitigate some of those costs? Before the current regulatory climate, they certainly used to. Keep in mind, too, that much of the high-touch clinical teaching is actually performed by residents and fellows, who are essentially paid for by the federal government and not by the institutions that employ them.

I am reminded by this interview with Robert Grossman, the NYU dean that made medical school free:

I mention that each year, N.Y.U.’s 450 medical students paid a total of $25 million in tuition.

“So where does this money go?” I ask Grossman.

“Well, where do you think?” he asks, smiling, raising his hands and shrugging shoulders. I think I know what he’s about to say, but I’m surprised when he says it so bluntly.

“It supports unproductive faculty,” he states coolly.

Unproductive faculty, Grossman explains, are people who draw a monthly paycheck, but don’t write grants, teach, or see patients. Tuition also funds other expenses, but the vast majority of tuition is not spent educating students.

All three—the cost, the price, and the debt—have run away and created a host of pathology within medicine. But if we are to take at face value that the price of medical school is merely a reflection of its soaring costs, then there is some serious fat to trim throughout every step of the process.

 

Student Loans Virtual Noon Conference

07.29.20 // Finance, Medicine, Radiology

I gave a virtual noon conference today for MRI Online. It requires a free registration, but it’s one of a collection of great radiology lectures available for free. This is week 19 of the series.

My talk is permalinked here. It starts with discussing a brief history of student loans in the US as well as a pretty detailed discussion of PSLF including dispelling some myths including an explanation of the high rejection rate.

If you listen and notice me laughing at the beginning, that’s because my Zoom session crashed when I attempted to share my screen and I had to restart. Audio cuts out here and there but is nearly 100% intact, pretty good for a Zoom call. And if you listen to any of my podcasts or other talks this past year, you can safely assume I’m sleep deprived (babies are cute) compounded today as I ended up covering the early morning 6 am shift, but it definitely has some really some useful nuggets for those who like audio/video. It’s no substitute, however, for sitting down for a few hours and reading my ad-free totally-free book in whatever format you choose.

One participant asked a great question that I incompletely answered during the Q&A at the end. It was, essentially, what happens to student loan debt after a divorce in a community property state like Texas? The answer is that it usually goes back to the individual borrower, but, that’s only because all assets and debts that happen before the marriage remain individual property and revert back to the individual while all things that happen during the marriage are shared equally. Since most people in the US have just undergraduate loans and most people get married after college, most people won’t have to deal with their spouse’s loans after a divorce. But certainly not all, and this is more likely to be an issue for doctors, who may enter school married or get married while in school. Timing is everything.

Panglossian Medical Fallacies

07.23.20 // Medicine

From Dr. Benjamin Mazer’s “Medicine’s dangerous optimism – Lessons from Dr. Pangloss,” published in The Journal of the Royal Society of Medicine.

Consider the story of Dr. Pangloss, the fictional “professor of metaphysico-theologico-cosmolo-nigology” in Voltaire’s satirical eighteenth-century novel Candide. Dr. Pangloss is remembered for declaring that we live in the “best of all possible worlds.” Pangloss could find logical explanations for the pain and turmoil he saw around him. No one suffered without a good reason. In the face of healthcare’s overwhelming complexity, doctors can also inadvertently resort to assuming our current situation is the best we can hope for.

I am defining Panglossian optimism as the unproven assumption that an observed outcome is the necessary outcome.

That’s the delightful set-up. He discusses four such fallacies:

  1. Favorable outcomes are attributable to medical care, unfavorable outcomes to a lack of it
  2. Arduous training and examination are what produce good doctors
  3. Physician outcomes predict patient outcomes
  4. A sufficiently popular intervention cannot be tested

I particularly love #2. It is, in part, the fallacy of hazing as a constructive and formative practice.

If evaluating applicants is currently too challenging without a Step 1 score, then a Step 1 score must be necessary to evaluate applicants. This is not a sound assumption. A tool that does not select for the qualities we desire inserts bias and noise into the process, making it less efficient.

We also encounter faulty Panglossian reasoning in debates over whether residency duty hours should be restricted for patient and trainee wellbeing. Many experienced physicians imagine their skills molded in the cauldron of inhumane work hours. It is true that they worked inhumane hours, and that many possess excellent skills. The Panglossian assumption is that the latter derives from the former.

For those dejected by the state of high-stakes exams, the arduousness of the journey to attendinghood, or the feebleness of so much of our tautological medical science, his conclusion:

We rationalize the irrational in times of perceived helplessness. By creating an environment more hospitable to questioning and change, we may be less drawn to false comforts.

So, when you are done, or, when you have a position of authority: be part of the solution.

Review: Orbit CME

07.20.20 // Medicine, Radiology, Reviews

It was always a good idea, but in this new world where conferences and live events are canceled for the foreseeable future, Orbit CME is a great idea.

(I previously got temporary free access to Orbit for the purposes of writing this review over a year ago, and I’ve got the usual reader discount affiliate link combo for you here: $20 off any plan. As there aren’t any ads here, these types of win-win situations for good products are one of the only ways I earn money through my writing. So there’s your COI disclosure.)

Orbit is a web browser plug-in that promises to automatically track and quantify the qualifying educational activities you do every day on your computer and then provide you with effortless legit AMA PRA Category 1 CME (often including the somewhat more challenging “self-assessment” SA-CME that some fields require that you typically get for answering questions during each lecture at a medical conference or other interactive activity).

How does it shape up?

Pretty darn well. In order to deliver the value of your subscription, you need to have the abilities/privileges to install the orbit browser plug-in (which is currently only available for Google Chrome). This plugin monitors your browsing and triggers whenever you visit a website that might come in handy for CME, like UpToDate, PubMed, Radiographics, or Radiopaedia. It measures your time with that active browser window, and generates an entry in your CME log. You can then choose which entries to actually spend a credit on to get the CME for it, in case you need certain types (like medical ethics in Texas or MQSA, Cardiac CT, fluoro, etc).

For example, while the hospital PCs only had internet explorer installed until recently, I had no problem using Chrome with the plugin when I worked at imaging centers or from home. As a radiologist, I earned CME so fast just from my usual day-to-day work that even if only using it on my home PC I would have been able to get the entire year’s worth of credits within a month or so, at which point I just uninstalled the plug-in.

Every once in a while I would have random difficulty logging into the plug-in (which does require logins periodically to make sure you’re still you), but otherwise, the process was completely seamless. CME is provided through Tufts, and you can download detailed CME logs for submission to various bodies that require such things.

You can also post external CME to the Orbit site allowing you to track all of your CME in one place and generate one report containing everything you’ve done. Very handy.

When I first discussed the product with the Orbit founding team back after finishing fellowship, I couldn’t help but feel that the price was too steep and thus not worth it ($360/yr for 25 credits; $600/2yr for 50 credits). But then I saw how truly effortless it was and how much a hassle the documentation burden of CME can be. If you enjoy conferences, it’ll always be possible to get enough CME through the activities you plan on pursuing anyway. If you’re in academics, you may acquire enough through your work activities like tumor boards and grand rounds to not need anything else.

But for those who don’t—and certainly in the current COVID world we live in where nothing is happening except remotely—I would rather pay to have my CME automatically generate itself than to do so via a virtual meeting (or some other laborious educational activity). I’ve got an infant and a preschooler, a busy practice, and a bunch of hobbies that are struggling for a minute of sunshine. I do CME every single day I work, and this gives me credit for that. Even if you get CME from other places like I do, there was something especially nice about not needing to bother tracking it down or keeping personal records because Orbit gives you everything you need anyway. If you have an academic/educational/CME fund, it’s definitely money well spent. It also works for PAs and NPs in addition to physicians.

The product was initially designed by a radiologist for radiologists, and it is absolutely perfectly suited to our workflow. But it also works well for many other specialties, and they have a handy table here telling you what kind of CME plugin can get you relative to the demands of your specialty society.

I’ll be subscribing again.

 

Explanations for the 2020 Official Step 1 Practice Questions

05.26.20 // Medicine

Another year, another set of explanations. As always, the order here reflects that of the new PDF released in February 2020. The official practice material page subsequently reverted back to the 2019 version, but as you can see the 2020 link remains live. (If you’re taking this in 2021, then see those explanations for updated questions #24 and 53)

Last year’s 2019 set is available here, though it was almost entirely a repeat of the 2018 set explained here.

The asterisks (*) signifies a new question, of which there are 36.

(more…)

The disappearing USMLE 2020 practice questions

05.20.20 // Medicine

Update: my explanations for these sets are now available: Step 1 and Step 2 CK.

Earlier this year the NBME released updated 2020 practice materials for both Step 1 and Step 2 CK. There are a bunch of new Step 1 questions and the Step 2 set is entirely brand new (the first meaningful change in several years). I was going to continue my annual tradition of explanations but then noticed that the website reverted back to the 2019 sets.

As is their tendency, the NBME isn’t very quick to manage outdated URLs. The PDFs for both new sets are still accessible:

I don’t know if these will simply come back as official sets soon, but it seems they were likely removed because of COVID-19, so it might be a while.

But they are free highest-quality questions and remain worth your time.