Lessons from the American Board of Surgery’s Virtual Board Exam Debacle

Lesson 1: Beware Third Party Services

During the slow drama that has been the delivery of high-stakes during the pandemic, some organizations like the American Board of Radiology resisted agreeing to a virtual exam for so long precisely because they couldn’t control the environment, delivery, and security. Unfortunately, we have some great evidence that offloading exam security to a third-party company can easily fail on those metrics as well.

Just ask the American Board of Surgery, which hired Verificient’s Proctortrack, an “industry leading Online Proctoring / Remote Invigilation solution.” Leaving aside how creepy “invigilation solution” sounds, Proctortrack has a 1-star rating on the app store for being generally awful. And despite the platform’s buzzwords of AI, computer vision, and machine learning, there were both serious mechanical and human failings.

The ABS went from a single day 8-hour exam to a two-day 4-hour exam, just to make sure the platform could handle it. It couldn’t.

ProtorTrack is extremely invasive. It requires download and installation of monitoring software that gives serious access over both your computer and your smartphone. The service also requires a room scan of 360 degrees to clear your workspace of nefarious intent (that last part actually makes sense to me).

So what happened?

Near instantaneous failure. The Room Scan feature failed early, and then examinees were basically left out to dry with poor phone and online customer service. People kept trying to log in and ended up banging their heads against the wall. I’ve heard an estimate that around half of the residents were able to take at least a portion of the test. But the circumstances were less than ideal, with many residents apparently reporting being harassed to complete distraction by the “live” overseas proctors.

Shortly after, the reports of identity theft, unauthorized credit card charges, and inappropriate social media contact began.

Because the test questions were exposed to a fraction of the examinees, those questions are no longer usable. The ABMS boards talk a lot about “their” intellectual property, and while these questions are written by volunteers, we shouldn’t demean the serious effort that goes into crafting and then vetting them. MCQ tests may not be a great measure of medical competence, but that doesn’t mean a lot doesn’t go into making one. Having to throw out a test is a serious setback.

I don’t know what goes into how these organizations make these sorts of operational decisions, but I don’t think it takes a clairvoyant to predict that ProctorTrack might have been a suboptimal choice.

While it might look good to an uninvolved observer, I don’t think a third-party off-site unaccountable organization is the right move for a high-stakes virtual exam. While at home exams require more effort, a residency-based in-person exam with live or focused virtual proctoring would be relatively straightforward. Regardless, video proctoring and screen recording don’t even need to be live, so long as all information is saved and can be reviewed.

It’s also hard to imagine a doctor throwing away their career just to copy some exam questions, which is really what we’re worried about here more than someone cheating to pass.

Lesson 2: Be Flexible

The ABS essentially forced residents to take the test (before they canceled it and told them they couldn’t). Taking a pass this year would use up an attempt and looks bad for programs.

One one hand, that doesn’t sound so bad if it all worked out. But it didn’t. And when things don’t work out, people’s lives are way harder. Dedicated study time, job starts, family planning–everything. Doctors build their lives and programs plan their schedules around these exams. When we live in tough times, it’s best to err on the side of accommodation.

Any board needs to have contingency plans in place that will accommodate a rapid administration to waiting candidates. Write more questions now and have them ready to deploy. Running out of questions after a failed attempt and needing to push things back indefinitely in order to write more is a terrible plan.

Lesson 3: Prioritize Communication

Announcements about the exam were done primarily through Twitter. Emails and website updates came universally after a multi-hour delay. Twitter is fine, but Twitter shouldn’t be used as a replacement for individualized communication. When it comes to a career-defining exam, people should be getting so many emails (or potentially opt-in text messages) that they almost find it annoying.

By the time the ABS finally published a FAQ on its website, days had passed.

When the Resident Association of the American College of Surgeons (RASACS) and ABS hosted a virtual town hall, it was announced within five hours of the event and then limited to in-advance sign-ups of only 400 people, a small fraction of the affected residents.

Take-Home

I don’t want to suggest that doing a virtual exam for the first time in history is a straightforward endeavor.

What I do think is clear is that an organization whose sole purpose is to create and administer tests needs to be nearly flawless on that execution. Offloading tasks to third parties, even the commercial testing centers so commonly used, has long resulted in a suboptimal and often degrading experience. Third-party proctoring is apparently no different. I appreciate that ABMS member boards are not tech companies and will always use contractors to accomplish a variety of tasks, but that does not absolve them of responsibility for the final product.

Lastly, communication and accountability shouldn’t be an afterthought. Just because residents have no power over their respective boards doesn’t mean they aren’t worthy of consideration and respect.

Buying Disability Insurance As a Medical Student

Let’s start by saying that I’m certainly not the only person on the internet that thinks it’s critical for all doctors to buy true own-occupation disability insurance that protects you in the event that you become disabled and can’t earn your full income. Your earning potential is too high and school is too expensive to not protect. There’s a good chance you’ve already (or will soon) hear about DI all the time because your eyeballs and attention are valuable and disability insurance agents sponsor a lot of podcasts and a bunch of blogs.

And that’s actually basically okay because that helps pay the bills for a lot of meaningful content out there; these folks are paid by the insurance companies (not you); and you need to use one to buy a policy anyway.

I bought my disability insurance policy toward the end of residency, but when I look back at my post-call exhausted driving, occupational hazards (e.g. sharps), and health scares, I feel fortunate to still be healthy and to have gone through the process unscathed. Though we haven’t needed to use it yet, it’s not hard to imagine a scenario where things didn’t pan out so well or where I or my wife developed a condition that prevented us from buying insurance in the future.

I have no doubt that the best answer to the question of “when to buy disability insurance?” is as soon as feasible. And I wondered more about the logistics of buying a physician policy as a medical student, something that no one really talks about.

So I asked Matt Wiggins from Pattern to talk with me and fill in the gaps. He made a video for you, dear reader (good for anyone but especially medical students and residents), and I’ve written a post breaking down how that works in medical school and detailing my thoughts. This isn’t a sponsored post (we don’t do those around here), but I do have a relationship with Pattern if you end up checking them out to get policy quotes. (It never costs anything to see your options; agents get paid a commission if you buy a policy.)

What is disability insurance?

A disability insurance policy will provide you with monthly paychecks if you become unable to carry out the duties of your job due to disability. The more you earn, the bigger a policy benefit you can purchase (and the more it costs in premiums). Disability insurance ensures that you are protected financially when things go wrong and you can’t work the way you used to.

A good policy for a doctor is called an own-occupation policy because it pays the full benefit if you can’t do the same doctoring job you were doing as a physician when you become disabled, even if you can be gainfully employed otherwise. If a surgeon hurts her hands and can’t operate, then she’s fully disabled even if she goes on to make even more money as a consultant or another kind of doctor. The problem with most policies bundled with your employment is 1) they don’t follow you when you leave your job and 2) they typically don’t have as strong a definition as own-occupation or what defines a disability. The practical matter is that a group policy just may not cover you in real life. The sorts of policies residents have while in training are notorious for letting you down when you need them most.

When you buy a policy, you will also choose from a variety of “riders,” which are essentially optional add-ons you can purchase a la carte. Each one makes your policy more expensive but also makes it more flexible. A common example would be a “future benefit increase” rider, which allows you to upgrade to a bigger policy in the future when your income rises without needing to go through medical underwriting. Even if you develop a medical condition that is sure to result in a disability in the future, the company still has to let you exercise the rider.

When is the right time to purchase disability insurance?

So you definitely need it. The question is just the timing. Ultimately, since you can’t predict the future, the real answer is as soon as you are eligible to buy the right kind of policy and can afford it. The first part has a real answer, the affordability part is a little fuzzier.

There are two long-term financial benefits to grabbing a policy early:

— Cheaper rates based on Age and Health (you’re never younger or probably healthier than you are right now)

— Discounts from University or Training-program affiliation (range anywhere from 10-40% off the premium and will typically last the life of your policy even after you leave)

For many, a good solution is to get a very small policy towards the end of medical school. You can lock in $1,000/month in coverage for $20-$40/month, which would at least provide $1,000/month tax-free until you are retirement age should a disability happen to you in medical school. But the main benefit is that this small purchase would lock you into the ability to increase your coverage to much higher levels without the insurance company ever checking on your health again, which can be the difference between being future-proof or not. It’s all about who you are when you buy, not who you become.

How Things Differ for a Medical Student

A resident or attending buying disability insurance is able to buy an own-occupation specialty-specific policy. This is the kind that protects exactly what you do. A pre-match medical student will instead get a generic (internal medicine) policy. If you buy a policy after matching, the rate will be adjusted for the risk category (and procedures etc) of the field they’ve chosen. So an anesthesiologist, a higher-risk specialty, will have a higher rate than a family doc.

But there is some nuance. I asked Matt how that works if you specialize later on, and this is what he said:

When a doctor files an own-occupation claim, the insurance company looks to find out exactly what duties or procedures the doctor is doing at the time of disability, and that is the occupation that is protected. In other words, if a doctor buys a policy in med school and then goes on to an internal medicine residency, followed by a cardiology residency, followed by an interventional cardiology fellowship, they will be protected for the procedures they are doing as an interventional cardiologist even though they bought their policy well before they were an interventional cardiologist and their rates are cheaper than if they had purchased the policy later as an attending IC doc.

Interesting (and not what I would have expected at all).

So, if you’ve chosen a higher risk field like a surgical specialty or anesthesiology, then you’ll also save money in the long term by getting the rates of a less risky “generic doctor” profession upfront.

Most companies will only offer policies to fourth-year students, who can purchase a benefit of up to $2500/month (which would cost in the neighborhood of $60-$100/month in premiums for men and $75-$125/month for women). Once you match, you’d be eligible to increase to the resident benefit (which is 5,000/month across the board). But you don’t have to buy a policy at the maximum you’re eligible for; you can buy one that you’re confident you can afford.

Parting Thoughts

The idea that I could have skipped a couple of burritos and a latte and locked in disability insurance as a medical student is crazy to me. I waited until late in residency when we had more cash flow, but that was I think the wrong approach. I should have purchased a small policy as soon as possible. Even if I couldn’t afford the higher premiums to increase the benefit until later in training, at least I could have guaranteed that flexibility to do so upfront by being more proactive.

 

 

The Cost, Price, and Debt of Medical Education

From “The Cost, Price, and Debt of Medical Education” in NEJM.

In the 1960s, 4 years of U.S. medical education could be purchased for about $40,000 (in 2018 dollars). By 2018, the average price had increased by 750%, to about $300,000; approximately 75% of students took on loans, and their average debt at graduation was $200,000. In contrast, U.S. college tuition increased by about 250% over the same period.

…between 2010 and 2018, the percentage of medical students graduating with no debt also increased — a happy result if it arose from lower prices or more scholarships, but its occurrence in the absence of those conditions suggests that medicine is increasingly a profession accessible only to the rich.

What a massive increase.

This last part, borne out by the data, is a serious problem. And a few free medical schools and a handful of scholarships isn’t going to reverse the serious financial headwinds discouraging the nonprivileged.

Of course, loan-repayment programs offer nothing to students who don’t finance their education with debt to begin with. But a more fundamental limitation is that because these programs target debt and not price or cost, they risk exacerbating high education prices for all. In theory, students who expect their loans to be fully or partially forgiven become less sensitive to the price of medical education and the price differences among schools. Anticipated debt relief — even partial or uncertain relief — reduces the already weak incentives for medical schools to compete on price and so effectively transfers money to the schools with higher prices.

Price insensitivity is a huge issue across higher education. The investment in a nebulous professional future paid with borrowed funds means that many students just accept the sticker price of whatever their dream is and only consider the consequences after the fact. For medical school, many students are just happy to be accepted. The cost differential amongst choices often only plays a role for those choosing between acceptances from lower-price home state institutions and higher-price private ones.

People often ask if PSLF will be canceled as Trump has proposed or if the government will start forgiving large swaths of student loans as many candidates proposed during the recent Democratic primaries, but that’s missing a key part of the story. Any changes to the details of loan repayment in the absence of an overhaul of higher education funding will not solve the systemic problem.

The cost of medical school–as in the amount of money it takes to train a medical student–is surely high. But is it really 3 times that of a college? Do medical students, in some situations, not also provide value to help mitigate some of those costs? Before the current regulatory climate, they certainly used to. Keep in mind, too, that much of the high-touch clinical teaching is actually performed by residents, who are essentially paid for by the federal government and not by the institutions that employ them.

I am reminded by this interview with Robert Grossman, the NYU dean that made medical school free:

I mention that each year, N.Y.U.’s 450 medical students paid a total of $25 million in tuition.

“So where does this money go?” I ask Grossman.

“Well, where do you think?” he asks, smiling, raising his hands and shrugging shoulders. I think I know what he’s about to say, but I’m surprised when he says it so bluntly.

“It supports unproductive faculty,” he states coolly.

Unproductive faculty, Grossman explains, are people who draw a monthly paycheck, but don’t write grants, teach, or see patients. Tuition also funds other expenses, but the vast majority of tuition is not spent educating students.

All three–the cost, the price, and the debt–have run away and created a host of pathology within medicine. But if we are to take at face value that the price of medical school is merely a reflection of its soaring costs, then there is some serious fat to trim throughout every step of the process.

 

Student Loans Virtual Noon Conference

I gave a virtual noon conference today for MRI Online. It requires a free registration, but it’s one of a collection of great radiology lectures available for free. This is week 19 of the series.

My talk is permalinked here. It starts with discussing a brief history of student loans in the US as well as a pretty detailed discussion of PSLF including dispelling some myths including an explanation of the high rejection rate.

If you listen and notice me laughing at the beginning, that’s because my Zoom session crashed when I attempted to share my screen and I had to restart. Audio cuts out here and there but is nearly 100% intact, pretty good for a Zoom call. And if you listen to any of my podcasts or other talks this past year, you can safely assume I’m sleep deprived (babies are cute) compounded today as I ended up covering the early morning 6 am shift, but it definitely has some really some useful nuggets for those who like audio/video. It’s no substitute, however, for sitting down for a few hours and reading my ad-free totally-free book in whatever format you choose.

One participant asked a great question that I incompletely answered during the Q&A at the end. It was, essentially, what happens to student loan debt after a divorce in a community property state like Texas? The answer is that it usually goes back to the individual borrower, but, that’s only because all assets and debts that happen before the marriage remain individual property and revert back to the individual while all things that happen during the marriage are shared equally. Since most people in the US have just undergraduate loans and most people get married after college, most people won’t have to deal with their spouse’s loans after a divorce. But certainly not all, and this is more likely to be an issue for doctors, who may enter school married or get married while in school. Timing is everything.

Panglossian Medical Fallacies

From Dr. Benjamin Mazer’s “Medicine’s dangerous optimism – Lessons from Dr. Pangloss,” published in The Journal of the Royal Society of Medicine.

Consider the story of Dr. Pangloss, the fictional “professor of metaphysico-theologico-cosmolo-nigology” in Voltaire’s satirical eighteenth-century novel Candide. Dr. Pangloss is remembered for declaring that we live in the “best of all possible worlds.” Pangloss could find logical explanations for the pain and turmoil he saw around him. No one suffered without a good reason. In the face of healthcare’s overwhelming complexity, doctors can also inadvertently resort to assuming our current situation is the best we can hope for.

I am defining Panglossian optimism as the unproven assumption that an observed outcome is the necessary outcome.

That’s the delightful set-up. He discusses four such fallacies:

  1. Favorable outcomes are attributable to medical care, unfavorable outcomes to a lack of it
  2. Arduous training and examination are what produce good doctors
  3. Physician outcomes predict patient outcomes
  4. A sufficiently popular intervention cannot be tested

I particularly love #2. It is, in part, the fallacy of hazing as a constructive and formative practice.

If evaluating applicants is currently too challenging without a Step 1 score, then a Step 1 score must be necessary to evaluate applicants. This is not a sound assumption. A tool that does not select for the qualities we desire inserts bias and noise into the process, making it less efficient.

We also encounter faulty Panglossian reasoning in debates over whether residency duty hours should be restricted for patient and trainee wellbeing. Many experienced physicians imagine their skills molded in the cauldron of inhumane work hours. It is true that they worked inhumane hours, and that many possess excellent skills. The Panglossian assumption is that the latter derives from the former.

For those dejected by the state of high-stakes exams, the arduousness of the journey to attendinghood, or the feebleness of so much of our tautological medical science, his conclusion:

We rationalize the irrational in times of perceived helplessness. By creating an environment more hospitable to questioning and change, we may be less drawn to false comforts.

So, when you are done, or, when you have a position of authority: be part of the solution.